Oct 31 03:06:05.691774 ip-10-0-140-194 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:06:06.151704 ip-10-0-140-194 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:06.151704 ip-10-0-140-194 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:06:06.151704 ip-10-0-140-194 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:06.151704 ip-10-0-140-194 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:06.155268 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155190    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.155268 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155268    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.155268 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155271    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155275    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155278    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155280    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155283    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155286    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155289    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155291    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155294    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155297    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155299    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155302    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155304    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155307    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155313    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155316    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155319    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155321    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155324    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155326    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.155368 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155329    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155331    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155333    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155336    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155338    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155341    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155344    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155346    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155348    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155351    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155354    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155356    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155359    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155362    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155365    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155367    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155370    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155373    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155376    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.155851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155379    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155381    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155384    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155386    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155389    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155392    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155394    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155397    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155399    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155402    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155405    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155407    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155410    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155412    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155415    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155417    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155420    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155422    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155425    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155427    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.156343 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155430    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155432    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155435    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155438    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155441    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155461    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155464    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155470    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155475    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155478    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155481    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155484    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155489    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155493    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155496    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155499    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155502    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155504    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155507    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.156820 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155509    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155512    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155514    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155516    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155519    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155521    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155524    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155526    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155529    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155531    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155534    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155536    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155539    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155541    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155544    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155546    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155549    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155552    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155555    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155557    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.157277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155561    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155564    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155567    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155569    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155572    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.155575    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156343    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156350    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156353    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156356    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156359    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156361    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156364    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156367    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156370    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156372    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156375    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156378    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156380    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.157764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156383    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156386    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156388    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156391    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156393    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156396    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156399    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156401    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156404    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156407    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156409    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156412    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156415    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156418    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156420    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156423    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156426    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156428    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156431    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156434    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.158218 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156436    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156439    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156441    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156464    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156469    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156473    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156477    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156481    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156483    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156486    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156489    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156491    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156494    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156500    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156503    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156505    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156508    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156510    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156513    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.158753 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156516    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156518    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156521    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156524    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156526    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156529    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156539    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156542    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156544    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156547    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156550    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156552    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156555    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156558    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156561    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156564    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156567    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156569    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156572    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156574    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.159215 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156577    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156581    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156584    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156587    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156589    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156594    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156602    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156605    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156608    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156611    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156614    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156616    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156619    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156621    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156625    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156629    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156632    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156634    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156637    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.159720 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156640    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156642    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156645    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156647    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156650    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156652    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156655    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156658    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156661    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156663    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156666    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156668    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156671    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156673    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.156676    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157160    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157165    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157168    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157171    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.160176 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157174    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157177    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157181    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157185    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157189    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157192    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157194    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157197    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157201    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157204    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157207    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157209    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157212    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157214    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157217    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157220    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157223    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157225    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157228    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157230    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.160642 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157233    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157236    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157239    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157241    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157244    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157247    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157249    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157252    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157255    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157257    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157259    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157262    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157265    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157267    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157270    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157273    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157275    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157278    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157281    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.161117 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157284    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157289    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157292    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157295    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157297    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157301    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157304    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157306    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157309    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157311    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157314    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157316    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157319    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157321    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157324    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157327    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157329    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157331    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157334    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157336    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.161590 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157339    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157341    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157343    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157346    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157348    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157351    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157353    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157356    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157359    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157361    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157363    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157366    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157369    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157373    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157375    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157378    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157380    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157383    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157385    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.162046 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157388    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157391    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157393    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157396    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157398    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157400    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157403    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157405    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157408    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157411    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157413    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157415    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157418    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157420    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157422    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157425    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157427    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157429    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157432    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157435    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.162524 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157437    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157440    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157458    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.157461    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157557    2570 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157565    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157573    2570 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157578    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157583    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157587    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157591    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157595    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157598    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157601    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157605    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157608    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157611    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157614    2570 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157617    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157620    2570 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157623    2570 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157627    2570 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157631    2570 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157635    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:06:06.163019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157638    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157641    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157644    2570 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157648    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157651    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157654    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157659    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157662    2570 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157665    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157668    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157671    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157674    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157678    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157681    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157686    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157689    2570 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157692    2570 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157695    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157699    2570 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157703    2570 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157706    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157709    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157712    2570 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157716    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:06:06.163629 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157719    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157722    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157725    2570 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157728    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157731    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157734    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157747    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157750    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157753    2570 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157755    2570 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157764    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157767    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157770    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157773    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157778    2570 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157781    2570 flags.go:64] FLAG: --help="false"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157784    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157787    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157790    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157793    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157796    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157800    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157804    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:06:06.164260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157807    2570 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157810    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157813    2570 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157815    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157818    2570 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157821    2570 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157824    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157827    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157830    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157833    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157836    2570 flags.go:64] FLAG: --lock-file=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157839    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157842    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157845    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157852    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157855    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157859    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157863    2570 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157865    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157869    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157871    2570 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157874    2570 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157878    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157883    2570 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157886    2570 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:06:06.164818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157889    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157892    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157895    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157898    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157901    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157904    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157908    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157914    2570 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157917    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157920    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157923    2570 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157926    2570 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157929    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157932    2570 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157935    2570 flags.go:64] FLAG: --port="10250"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157938    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157941    2570 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0681b6506b7fbaf66"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157944    2570 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157947    2570 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157950    2570 flags.go:64] FLAG: --register-node="true"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157953    2570 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157957    2570 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157960    2570 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157962    2570 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157968    2570 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157972    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:06:06.165431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157975    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157978    2570 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157981    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157983    2570 flags.go:64] FLAG: --runonce="false"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157987    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157990    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157993    2570 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157996    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.157999    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158002    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158005    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158008    2570 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158012    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158015    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158018    2570 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158021    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158024    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158027    2570 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158030    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158035    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158038    2570 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158041    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158045    2570 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158047    2570 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158050    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:06:06.166044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158053    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158056    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158058    2570 flags.go:64] FLAG: --v="2"
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158063    2570 flags.go:64] FLAG: --version="false"
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158068    2570 flags.go:64] FLAG: --vmodule=""
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158071    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158075    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158207    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158211    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158215    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158220    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158225    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158229    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158233    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158236    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158239    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158242    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158245    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158248    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158252    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.166669 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158254    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158257    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158259    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158262    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158264    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158267    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158269    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158272    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158274    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158277    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158279    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158282    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158284    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158287    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158289    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158292    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158294    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158297    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158299    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.167155 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158302    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158304    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158307    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158309    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158313    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158316    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158318    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158321    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158323    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158326    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158328    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158331    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158333    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158336    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158338    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158341    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158343    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158345    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158348    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158350    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.167731 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158353    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158355    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158358    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158360    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158362    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158365    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158367    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158369    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158372    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158374    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158377    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158379    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158381    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158384    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158387    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158389    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158393    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158396    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158398    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158401    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.168205 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158403    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158406    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158408    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158411    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158413    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158416    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158418    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158420    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158423    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158425    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158427    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158430    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158433    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158435    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158438    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158440    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158457    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158459    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158462    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158464    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.168680 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158467    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158469    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158472    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158474    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158477    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158479    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158482    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158485    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158489    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158491    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158494    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158496    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158499    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158502    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.158528    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158564    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.169148 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158568    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158571    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158573    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158576    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158579    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158581    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158584    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158586    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158589    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158592    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158594    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158596    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158599    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158601    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158604    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158606    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158609    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158611    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158614    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.169557 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158616    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158619    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158621    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158624    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158627    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158631    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158633    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158636    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158638    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158640    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158644    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158646    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158649    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158652    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158655    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158657    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158660    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158662    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158665    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158667    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.170105 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158670    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158672    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158675    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158677    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158679    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158682    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158684    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158687    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158690    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158692    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158695    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158697    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158700    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158702    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158705    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158707    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158710    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158714    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158717    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158719    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.170615 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158721    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158724    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158728    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158731    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158733    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158736    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158738    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158741    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158744    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158746    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158749    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158751    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158754    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158756    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158759    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158762    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158764    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158767    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158769    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.171091 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158772    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158775    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158778    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158780    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158782    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158785    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158789    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158792    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158794    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158797    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158801    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158805    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158809    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158812    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158816    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158819    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158822    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158825    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158828    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.171637 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158831    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158833    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158836    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158838    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158841    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158843    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158846    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.158848    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.166323    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.166338    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.167076    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.167082    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167168    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167173    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167177    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167180    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167182    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167185    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167187    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.172073 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167191    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167194    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167196    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167199    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167201    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167210    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167213    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167216    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167218    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167221    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167223    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167228    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167232    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167235    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167238    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167241    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167243    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167246    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167248    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.172848 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167251    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167253    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167257    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167259    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167262    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167265    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167267    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167270    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167272    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167275    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167277    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167280    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167282    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167285    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167287    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167290    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167292    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167294    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167297    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167307    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.173294 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167310    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167313    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167315    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167318    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167320    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167322    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167325    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167327    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167330    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167332    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167335    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167337    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167340    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167342    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167345    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167348    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167350    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167353    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167356    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.173817 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167359    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167361    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167363    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167367    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167371    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167374    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167376    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167379    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167381    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167383    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167386    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167388    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167391    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167400    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167403    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167405    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167408    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167410    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167413    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.174255 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167415    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167418    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167421    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167424    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167426    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167429    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167431    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167434    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167437    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167440    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167460    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167465    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167468    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167471    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167473    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167475    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167478    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167481    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167483    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167486    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.174741 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167488    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167491    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.167519    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167706    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167711    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167713    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167716    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167725    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167728    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167731    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167733    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167736    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167740    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167742    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167745    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167748    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.175248 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167751    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167753    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167756    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167758    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167761    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167764    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167766    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167769    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167771    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167774    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167777    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167779    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167781    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167784    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167786    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167788    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167791    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167793    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167796    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167799    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.175839 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167801    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167804    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167806    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167809    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167818    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167822    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167826    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167829    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167832    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167834    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167837    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167839    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167842    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167844    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167847    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167856    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167858    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167861    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167863    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.176441 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167866    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167868    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167871    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167873    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167876    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167878    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167880    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167883    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167885    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167888    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167890    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167893    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167895    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167897    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167900    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167902    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167905    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167907    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167917    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167920    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.176895 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167922    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167925    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167929    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167932    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167935    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167938    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167940    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167942    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167945    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167947    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167950    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167952    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167955    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167958    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167960    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167962    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167965    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167967    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167970    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.177388 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167972    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167975    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167977    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167980    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167982    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167984    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167987    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167990    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167992    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167995    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.167997    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168000    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168008    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168011    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168014    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.178050 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.168037    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168081    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168085    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168087    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168090    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168092    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168095    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168098    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168101    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168103    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168106    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168108    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168110    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168113    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168115    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168118    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168120    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168122    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168125    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168127    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:06.178645 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168130    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168132    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168134    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168137    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168139    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168142    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168144    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168146    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168149    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168151    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168160    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168163    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168165    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168168    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168170    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168173    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168175    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168179    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168183    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:06.179277 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168186    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168189    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168191    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168194    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168196    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168199    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168201    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168203    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168206    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168208    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168211    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168214    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168216    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168219    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168221    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168224    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168226    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168229    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168231    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168233    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:06.179745 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168236    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168239    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168241    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168244    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168252    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168255    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168257    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168260    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168263    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168265    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168267    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168270    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168272    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168275    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168279    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168282    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168285    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168287    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168290    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:06.180228 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168292    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168295    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168297    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168300    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168302    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168308    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168310    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168313    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168315    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168317    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168320    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168323    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168325    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168327    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168332    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168334    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168337    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168339    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168349    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:06.180697 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168352    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168354    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168357    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168359    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168363    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168365    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168368    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168370    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168373    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:06.168375    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.168423    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.171002    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.172243    2570 server.go:1014] "Starting client certificate rotation"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.172349    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.172396    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.174737    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:06.181735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.179817    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:06.189084 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.189070    2570 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:06:06.194269 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.194138    2570 log.go:25] "Validated CRI v1 image API"
Oct 31 03:06:06.195339 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.195313    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:06:06.198011 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.197989    2570 fs.go:135] Filesystem UUIDs: map[40e51575-7ee4-4bdf-b66f-a3b87d546dc7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a4de9143-c1d0-4490-a9b2-79605ef3b55e:/dev/nvme0n1p3]
Oct 31 03:06:06.198074 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.198011    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 03:06:06.203818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.203706    2570 manager.go:217] Machine: {Timestamp:2026-10-31 03:06:06.202177723 +0000 UTC m=+0.381367100 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100023 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f04d97c0d665ac2f20c0c7a07bad SystemUUID:ec28f04d-97c0-d665-ac2f-20c0c7a07bad BootID:80283683-6a38-4ffb-9591-371bde0aceb7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fd:96:65:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fd:96:65:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:c0:d4:91:5f:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:06:06.203818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.203819    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:06:06.203931 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.203901    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:06:06.207425 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.207402    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:06:06.207593 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.207427    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:06:06.207640 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.207599    2570 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:06:06.207640 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.207607    2570 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:06:06.207640 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.207636    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:06.208494 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.208484    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:06.209867 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.209858    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:06:06.214613 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.214601    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:06:06.214732 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.214723    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:06:06.215545 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215535    2570 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:06:06.215596 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215549    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:06:06.215596 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215563    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:06:06.215596 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215575    2570 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:06:06.215596 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215587    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:06:06.215982 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.215966    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:06.218504 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.218485    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:06:06.226039 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.226025    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:06.226920 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.226908    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:06:06.226964 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.226939    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:06:06.227127 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227115    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:06:06.227157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227134    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:06:06.227157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227140    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:06:06.227157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227147    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:06:06.227157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227153    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:06:06.227264 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227159    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:06:06.227774 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227765    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:06:06.227808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227775    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:06:06.227808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227782    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:06:06.227808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227788    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:06:06.227808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227797    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:06:06.227808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.227806    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:06:06.228492 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.228482    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:06:06.228522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.228496    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:06:06.232652 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.232631    2570 server.go:1263] "Started kubelet"
Oct 31 03:06:06.232744 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.232675    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:06:06.236954 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.232794    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:06:06.236954 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.236813    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:06:06.237071 ip-10-0-140-194 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:06:06.239352 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.238429    2570 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:06:06.239352 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.238527    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:06:06.242620 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.242600    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 03:06:06.246719 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.246699    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:06.247375 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.247355    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:06:06.247534 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.247496    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.248121 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248106    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:06:06.248204 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248124    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:06:06.248204 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248108    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:06:06.248318 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248207    2570 factory.go:55] Registering systemd factory
Oct 31 03:06:06.248318 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248223    2570 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:06:06.248402 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248363    2570 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:06:06.248402 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248386    2570 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:06:06.248402 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248400    2570 factory.go:153] Registering CRI-O factory
Oct 31 03:06:06.248555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248408    2570 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:06:06.248555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248465    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:06:06.248555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248481    2570 factory.go:103] Registering Raw factory
Oct 31 03:06:06.248555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.248492    2570 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:06:06.249659 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.249607    2570 manager.go:319] Starting recovery of all containers
Oct 31 03:06:06.260875 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.260862    2570 manager.go:324] Recovery completed
Oct 31 03:06:06.264459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.264435    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.266235 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266217    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.266299 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266247    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.266299 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266258    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.266745 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266732    2570 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:06:06.266745 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266742    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:06:06.266876 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266758    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:06:06.266876 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.266765    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.267554 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.267537    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.267637 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.267565    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.267637 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.267575    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.268982 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.268970    2570 policy_none.go:50] "Start"
Oct 31 03:06:06.269027 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.268987    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:06:06.269027 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.268998    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:06:06.271009 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.270998    2570 policy_none.go:44] "Start"
Oct 31 03:06:06.314060 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.314019    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:06:06.316543 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316464    2570 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:06:06.316543 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.316500    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:06:06.316543 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316511    2570 server.go:87] "Starting device plugin registration server"
Oct 31 03:06:06.316787 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316774    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:06:06.316833 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316793    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:06:06.316880 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316851    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:06:06.316987 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316962    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:06:06.316987 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.316974    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:06:06.317526 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.317510    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:06:06.317600 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.317549    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.370508 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.363546    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-194.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 03:06:06.370508 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.363554    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-194.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 03:06:06.370508 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.363531    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-194.us-west-2.compute.internal.18e37e51c1cd10a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:06.232481953 +0000 UTC m=+0.411671330,LastTimestamp:2026-10-31 03:06:06.232481953 +0000 UTC m=+0.411671330,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-194.us-west-2.compute.internal,}"
Oct 31 03:06:06.386816 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.386787    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:06:06.388152 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.388138    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:06:06.388207 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.388158    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:06:06.388207 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.388178    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:06:06.388291 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.388208    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 03:06:06.417102 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.417045    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.417878 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.417860    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.417963 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.417890    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.417963 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.417902    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.417963 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.417930    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.427334 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.427254    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-194.us-west-2.compute.internal.18e37e51c3d01253  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-194.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:06.266233427 +0000 UTC m=+0.445422803,LastTimestamp:2026-10-31 03:06:06.266233427 +0000 UTC m=+0.445422803,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-194.us-west-2.compute.internal,}"
Oct 31 03:06:06.427683 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.427655    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-194.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 03:06:06.427759 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.427655    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 03:06:06.427759 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.427667    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 03:06:06.429555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.429540    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:06.429606 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.429595    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9skrb"
Oct 31 03:06:06.488758 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.488720    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"]
Oct 31 03:06:06.488869 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.488793    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.489703 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.489686    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.489782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.489713    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.489782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.489724    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.491951 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.491939    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.492116 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492100    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.492180 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492147    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.492687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492671    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.492795 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492696    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.492795 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492713    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.492795 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492715    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.492795 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492737    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.492795 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.492750    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.494877 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.494860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.494951 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.494893    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:06.495650 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.495634    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:06.495735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.495663    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:06.495735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.495679    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:06.512776 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.512759    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.514363 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.514343    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:06.514802 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.514786    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.520933 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.520919    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.520990 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.520939    2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-194.us-west-2.compute.internal\": node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.550713 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.550687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.550804 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.550714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.550804 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.550732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.622902 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.622867    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.651240 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651213    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.651329 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.651329 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651263    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.651412 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651327    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.651412 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651328    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.651412 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.651337    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.723794 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.723720    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.816453 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.816422    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.818567 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:06.818534    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:06.824278 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.824137    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:06.925024 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:06.924989    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:07.025572 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:07.025491    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:07.126338 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:07.126310    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:07.172138 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.172106    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 03:06:07.172606 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.172281    2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:06:07.172606 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:07.172273    2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9b46fe1a16de407895db29e9b878039-acea73c2d7879f85.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-194.us-west-2.compute.internal.18e37e51c3d0820b\": read tcp 10.0.140.194:49740->18.189.77.5:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-194.us-west-2.compute.internal.18e37e51c3d0820b  default   4888 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-194.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:06 +0000 UTC,LastTimestamp:2026-10-31 03:06:06.489727864 +0000 UTC m=+0.668917239,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-194.us-west-2.compute.internal,}"
Oct 31 03:06:07.226581 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:07.226556    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:07.326811 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:07.326746    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:06:07.376662 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.376641    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:07.433987 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.433954    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:07 +0000 UTC" deadline="2028-07-04 08:43:55.388017105 +0000 UTC"
Oct 31 03:06:07.433987 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.433981    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14693h37m47.954040687s"
Oct 31 03:06:07.448274 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.448254    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:07.506433 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.506408    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:07.507582 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.507568    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:06:07.562004 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:07.561967    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:08.732199 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:08.732172    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:09.013596 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.013559    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd102acf3d8823a0d171d30a06e49c39c.slice/crio-d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac WatchSource:0}: Error finding container d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac: Status 404 returned error can't find the container with id d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac
Oct 31 03:06:09.014031 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.014007    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d001e32831032e1eb4e57b6d85f040b.slice/crio-00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df WatchSource:0}: Error finding container 00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df: Status 404 returned error can't find the container with id 00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df
Oct 31 03:06:09.019733 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.019715    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:06:09.110752 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.110719    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:09.144956 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.144931    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:09.251613 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.251587    2570 apiserver.go:52] "Watching apiserver"
Oct 31 03:06:09.282687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.282627    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:09.347592 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.347563    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:06:09.347867 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.347849    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-r8762","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd","openshift-multus/network-metrics-daemon-s5fjf","openshift-network-diagnostics/network-check-target-2nrm8","openshift-cluster-node-tuning-operator/tuned-c72n4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-m9b2j","openshift-network-operator/iptables-alerter-brjlb","openshift-ovn-kubernetes/ovnkube-node-w9hwd","kube-system/konnectivity-agent-qhzmz","openshift-dns/node-resolver-mlv4n","openshift-image-registry/node-ca-9dj5j","kube-system/global-pull-secret-syncer-bj5qd","kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"]
Oct 31 03:06:09.351670 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.351646    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.353679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.353657    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.353776 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.353757    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:09.355816 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.355798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.355880 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.355863    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:09.357947 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.357930    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:09.358029 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.357992    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:09.360047 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.360028    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.362142 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.362125    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.364290 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.364270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.366394 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.366379    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.368608 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.368591    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.370920 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.370902    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.373174 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.373158    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.375438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.375417    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.384174 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.384160    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:06:09.396170 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wd6\" (UniqueName: \"kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.396251 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw64p\" (UniqueName: \"kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.396251 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.396360 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396360 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396360 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396337    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396466 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkv7s\" (UniqueName: \"kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396466 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396451    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.396562 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97wg\" (UniqueName: \"kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.396562 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.396643 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.396643 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nqz\" (UniqueName: \"kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.396643 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396616    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.396792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.396792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.396792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396750    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.396792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.396929 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396819    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxfp\" (UniqueName: \"kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.396929 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396929 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396884    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.396929 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.397080 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.397080 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.396968    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.397080 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:09.397080 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397062    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.397191 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.397191 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.397191 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.397461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.397577 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtvl\" (UniqueName: \"kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.397681 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.397681 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.397809 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397696    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.397809 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.397809 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397801    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.397957 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397832    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.397957 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397889    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.397957 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.398089 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.397988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.398089 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.398200 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.398200 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.398200 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398167    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.398317 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398221    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.398317 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.398317 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398302    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.398461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmhz\" (UniqueName: \"kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.398461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.398461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.398644 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.398644 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.398644 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398613    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.398836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398644    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.398836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398728    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.398836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.398836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.398836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398928    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.398989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.399076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399137    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.399404 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqzr\" (UniqueName: \"kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399600    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.399872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.399634    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.400802 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400779    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.400802 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400787    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:06:09.400967 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400813    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:06:09.400967 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400783    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:06:09.400967 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400850    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.400967 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.400932    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:06:09.408557 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.408497    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.408674 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.408594    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.408753 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.408738    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:06:09.410113 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.410100    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:06:09.410389 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.410373    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.410389 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.410375    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.427223 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.427188    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"d102acf3d8823a0d171d30a06e49c39c","Type":"ContainerStarted","Data":"d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac"}
Oct 31 03:06:09.428114 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.428093    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerStarted","Data":"00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df"}
Oct 31 03:06:09.442824 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.442771    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:06:09.442956 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.442936    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.443043 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.442986    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:06:09.443324 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443307    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.443324 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443310    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:06:09.443477 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443402    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:06:09.443477 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443440    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.443477 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443454    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:06:09.443477 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443471    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443468    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443504    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443511    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443507    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443555    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443560    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443593    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:06:09.443679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443610    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.443889 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.443687    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:06:09.446621 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446605    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:06:09.446687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446626    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:06:09.446687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446638    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:06:09.446687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446649    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:06:09.446687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446660    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:06:09.446687 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446680    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:06:09.446828 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446704    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:09.446828 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.446753    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfnjc"
Oct 31 03:06:09.500863 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500835    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500865    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500915    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtvl\" (UniqueName: \"kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500955    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500961    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.500990 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500958    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.500958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501001    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501023    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501133    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501168    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501216    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501184    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501234    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.501298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501279    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501315    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501325    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501355    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501354    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501380    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmhz\" (UniqueName: \"kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501456    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501546    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501555    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.502141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501571    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501585    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.501574    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501627    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501644    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.501673    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:10.00165883 +0000 UTC m=+3.145766239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501810    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501881    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501887    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501904    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501915    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.502939 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501948    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501913    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.501978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502025    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502011    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502053    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502059    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502076    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502109    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502123    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502153    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502147    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.503792 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502346    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqzr\" (UniqueName: \"kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502388    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502445    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502479    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502490    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502497    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502514    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wd6\" (UniqueName: \"kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw64p\" (UniqueName: \"kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502673    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.504522 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502691    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502697    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502722    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502744    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv7s\" (UniqueName: \"kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502772    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502784    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502795    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97wg\" (UniqueName: \"kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502822    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502870    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nqz\" (UniqueName: \"kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502894    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502938    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.502994    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503000    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.505160 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503024    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxfp\" (UniqueName: \"kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503049    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503084    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503139    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503187    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503250    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503295    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503400    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.503415    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:09.505696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.503462    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:10.003445594 +0000 UTC m=+3.147553015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503506    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503509    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503586    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503596    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503620    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503638    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503641    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503655    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503691    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503734    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503767    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503917    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503932    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503961    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503986    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506144 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.503990    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.504014    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.504133    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.504222    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.504981    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.505658    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.505811    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.505925    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.506615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.506051    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.555091 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.555026    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:09.555091 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.555060    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:09.555091 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.555075    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:09.555306 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:09.555151    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:10.055132119 +0000 UTC m=+3.199239539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:09.555306 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.555212    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtvl\" (UniqueName: \"kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.555589 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.555562    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmhz\" (UniqueName: \"kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.556677 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.556653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw64p\" (UniqueName: \"kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:09.557038 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557017    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nqz\" (UniqueName: \"kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.557126 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557060    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqzr\" (UniqueName: \"kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.557168 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557146    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97wg\" (UniqueName: \"kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.557548 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv7s\" (UniqueName: \"kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.557665 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557651    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wd6\" (UniqueName: \"kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.557730 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.557714    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxfp\" (UniqueName: \"kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.675217 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.675180    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:06:09.676469 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.676447    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:06:09.680495 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.680476    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:06:09.685499 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.685473    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc279cb7_148d_4787_983c_ea212b73a3cc.slice/crio-a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a WatchSource:0}: Error finding container a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a: Status 404 returned error can't find the container with id a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a
Oct 31 03:06:09.688076 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.688048    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff779f60_ee44_48e3_b1fa_777374b9788b.slice/crio-0a8467fdbb6162a8f84a8f45119b67e6c61b71faa6df9862f193f2b78a873444 WatchSource:0}: Error finding container 0a8467fdbb6162a8f84a8f45119b67e6c61b71faa6df9862f193f2b78a873444: Status 404 returned error can't find the container with id 0a8467fdbb6162a8f84a8f45119b67e6c61b71faa6df9862f193f2b78a873444
Oct 31 03:06:09.692395 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.692315    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8762"
Oct 31 03:06:09.700764 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.700745    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c7f2a8_6907_4ca4_b636_e0807b250ef9.slice/crio-965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897 WatchSource:0}: Error finding container 965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897: Status 404 returned error can't find the container with id 965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897
Oct 31 03:06:09.703702 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.703687    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:06:09.705841 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.705825    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:09.708297 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.708280    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:09.710276 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.710260    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:06:09.712606 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:09.712589    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:06:09.715308 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.715289    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f85eaf9_d372_4e12_8016_0ade17b5badc.slice/crio-5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2 WatchSource:0}: Error finding container 5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2: Status 404 returned error can't find the container with id 5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2
Oct 31 03:06:09.716264 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.716240    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984c8e10_fc2b_4012_9b5b_78b9e3770178.slice/crio-d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82 WatchSource:0}: Error finding container d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82: Status 404 returned error can't find the container with id d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82
Oct 31 03:06:09.718953 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.718921    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dcdbcd_d8e5_4400_880b_c89f3362af1b.slice/crio-4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea WatchSource:0}: Error finding container 4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea: Status 404 returned error can't find the container with id 4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea
Oct 31 03:06:09.724952 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.724927    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801996f5_e5f7_4ef2_93b0_ce1b36c94343.slice/crio-fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956 WatchSource:0}: Error finding container fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956: Status 404 returned error can't find the container with id fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956
Oct 31 03:06:09.725799 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:09.725775    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b1e5a2_6307_4472_9502_5e97c55ac76f.slice/crio-d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463 WatchSource:0}: Error finding container d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463: Status 404 returned error can't find the container with id d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463
Oct 31 03:06:10.007711 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.007684    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:10.007862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.007722    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:10.007862 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.007826    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:10.007862 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.007828    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:10.007961 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.007874    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:11.007862081 +0000 UTC m=+4.151969491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:10.007961 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.007888    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:11.007880666 +0000 UTC m=+4.151988062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:10.108129 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.108096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:10.108261 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.108239    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:10.108261 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.108257    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:10.108346 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.108265    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:10.108346 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.108312    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:11.10829709 +0000 UTC m=+4.252404487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:10.424428 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.424341    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:10.425029 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:10.424496    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:10.432063 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.431979    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerStarted","Data":"a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a"}
Oct 31 03:06:10.433843 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.433814    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlv4n" event={"ID":"d9b1e5a2-6307-4472-9502-5e97c55ac76f","Type":"ContainerStarted","Data":"d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463"}
Oct 31 03:06:10.435616 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.435590    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9dj5j" event={"ID":"801996f5-e5f7-4ef2-93b0-ce1b36c94343","Type":"ContainerStarted","Data":"fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956"}
Oct 31 03:06:10.437609 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.437550    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd" event={"ID":"4f85eaf9-d372-4e12-8016-0ade17b5badc","Type":"ContainerStarted","Data":"5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2"}
Oct 31 03:06:10.440097 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.440073    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8762" event={"ID":"78c7f2a8-6907-4ca4-b636-e0807b250ef9","Type":"ContainerStarted","Data":"965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897"}
Oct 31 03:06:10.441645 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.441627    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brjlb" event={"ID":"5fb14716-1ea1-40f5-9ac7-7476c9daa35b","Type":"ContainerStarted","Data":"2654e96dcd4b70e73a1fd1beca3ab3a9c833bc686b30a9079993f16cfe369fe3"}
Oct 31 03:06:10.443238 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.443214    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c72n4" event={"ID":"ff779f60-ee44-48e3-b1fa-777374b9788b","Type":"ContainerStarted","Data":"0a8467fdbb6162a8f84a8f45119b67e6c61b71faa6df9862f193f2b78a873444"}
Oct 31 03:06:10.444632 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.444609    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea"}
Oct 31 03:06:10.446215 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.446192    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhzmz" event={"ID":"984c8e10-fc2b-4012-9b5b-78b9e3770178","Type":"ContainerStarted","Data":"d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82"}
Oct 31 03:06:10.449769 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.449722    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:09 +0000 UTC" deadline="2028-04-10 21:11:57.839391792 +0000 UTC"
Oct 31 03:06:10.449769 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:10.449769    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12666h5m47.389626394s"
Oct 31 03:06:11.015559 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.015502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:11.015715 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.015583    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:11.015787 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.015775    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:11.015883 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.015840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:13.015820795 +0000 UTC m=+6.159928204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:11.015973 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.015960    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:11.016027 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.016002    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:13.015991349 +0000 UTC m=+6.160098760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:11.116675 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.116642    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:11.116849 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.116816    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:11.116849 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.116838    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:11.116946 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.116870    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:11.116946 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.116925    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:13.11690821 +0000 UTC m=+6.261015612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:11.426969 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.426935    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:11.427615 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.427092    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:11.427615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.427209    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:11.427615 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:11.427363    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:11.450185 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.450018    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:09 +0000 UTC" deadline="2028-06-28 23:43:44.719104989 +0000 UTC"
Oct 31 03:06:11.450185 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:11.450070    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14564h37m33.269038413s"
Oct 31 03:06:12.424487 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:12.424457    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:12.424675 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:12.424609    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.033509    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.033816    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.033958    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.034019    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:17.033999938 +0000 UTC m=+10.178107348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.034079    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:13.034173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.034112    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:17.034101709 +0000 UTC m=+10.178209119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:13.135061 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.134759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:13.135061 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.134927    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:13.135061 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.134947    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:13.135061 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.134959    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:13.135061 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.135021    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:17.135001288 +0000 UTC m=+10.279108707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:13.426980 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.426474    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:13.426980 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.426630    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:13.426980 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.426697    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:13.426980 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:13.426765    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:13.473134 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.473007    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="ab5ac43ac6f41c846f7f1ce51bc27546fcc18a8adc0f617c3676ebb67ab9428b" exitCode=0
Oct 31 03:06:13.473134 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.473080    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"ab5ac43ac6f41c846f7f1ce51bc27546fcc18a8adc0f617c3676ebb67ab9428b"}
Oct 31 03:06:13.484143 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.484110    2570 generic.go:356] "Generic (PLEG): container finished" podID="9d001e32831032e1eb4e57b6d85f040b" containerID="00e4baa2d2740ef7da6062bdcfb04b69cd973b9fc1def1871d6b4af2c18876ed" exitCode=0
Oct 31 03:06:13.484242 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:13.484159    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerDied","Data":"00e4baa2d2740ef7da6062bdcfb04b69cd973b9fc1def1871d6b4af2c18876ed"}
Oct 31 03:06:14.424365 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:14.424191    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:14.424365 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:14.424316    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:14.489302 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:14.488593    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"d102acf3d8823a0d171d30a06e49c39c","Type":"ContainerStarted","Data":"35df32fda1ad850d5f9f1315e426a04ffab1332bedaf5d5eefced371f6a29dc7"}
Oct 31 03:06:14.493102 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:14.492547    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerStarted","Data":"efb645d19e2e73f66246bcdd1e93448a518ee267725dfa4cab6f57f82137b2a2"}
Oct 31 03:06:14.698895 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:14.698831    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal" podStartSLOduration=6.698806961 podStartE2EDuration="6.698806961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:14.593924592 +0000 UTC m=+7.738032007" watchObservedRunningTime="2026-10-31 03:06:14.698806961 +0000 UTC m=+7.842914381"
Oct 31 03:06:15.426976 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:15.426943    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:15.427415 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:15.427143    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:15.427415 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:15.427328    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:15.427554 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:15.427441    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:15.501956 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:15.501890    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhzmz" event={"ID":"984c8e10-fc2b-4012-9b5b-78b9e3770178","Type":"ContainerStarted","Data":"3bf05a1daa63c971e8bcf34d737317c4099a54605c76766ba45a514083c7b533"}
Oct 31 03:06:15.610168 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:15.610102    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" podStartSLOduration=7.610086102 podStartE2EDuration="7.610086102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:14.699179584 +0000 UTC m=+7.843287005" watchObservedRunningTime="2026-10-31 03:06:15.610086102 +0000 UTC m=+8.754193520"
Oct 31 03:06:16.424380 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:16.424347    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:16.424572 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:16.424485    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:16.507843 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:16.507109    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9dj5j" event={"ID":"801996f5-e5f7-4ef2-93b0-ce1b36c94343","Type":"ContainerStarted","Data":"1994ea4cf252a1075710848fc2bfe491f4736af1caa567773ea757a9b7c4b3a7"}
Oct 31 03:06:16.511203 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:16.511031    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd" event={"ID":"4f85eaf9-d372-4e12-8016-0ade17b5badc","Type":"ContainerStarted","Data":"c6c897aa0a8cdf3cab9f2d0d7dc84e5a08c3ac2149725e61d8db45b9dee06934"}
Oct 31 03:06:16.614040 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:16.613990    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhzmz" podStartSLOduration=4.721257823 podStartE2EDuration="9.613969697s" totalImagesPullingTime="4.892711874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:15.611318893 +0000 UTC m=+8.755426312" watchObservedRunningTime="2026-10-31 03:06:16.613969697 +0000 UTC m=+9.758077116"
Oct 31 03:06:17.070191 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:17.070145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:17.070367 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:17.070219    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:17.070433 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.070378    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:17.070468 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.070444    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:25.070425504 +0000 UTC m=+18.214532924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:17.070674 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.070650    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:17.070760 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.070718    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:25.07070322 +0000 UTC m=+18.214810628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:17.171586 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:17.171548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:17.171810 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.171782    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:17.171810 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.171811    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:17.171961 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.171824    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:17.171961 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.171884    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:25.171864849 +0000 UTC m=+18.315972288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:17.424928 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:17.424853    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:17.425074 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.424969    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:17.425420 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:17.425395    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:17.425547 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:17.425509    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:18.424428 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.424392    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:18.424941 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:18.424513    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:18.519431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.519390    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerStarted","Data":"c7a8b8a4cd5bb8bbd40bc51afe3351070d9990beba0b9bc10296cae46a563607"}
Oct 31 03:06:18.522292 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.522262    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mlv4n" event={"ID":"d9b1e5a2-6307-4472-9502-5e97c55ac76f","Type":"ContainerStarted","Data":"0f7e79d5cb0f504d2493f568e55b84e3741646daaf056236a05195f3d438bdbd"}
Oct 31 03:06:18.525071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.525043    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8762" event={"ID":"78c7f2a8-6907-4ca4-b636-e0807b250ef9","Type":"ContainerStarted","Data":"da27eec94b8f8cfda507ca4da30067f4e8ccb059cbf82362a5fbd1330461f82d"}
Oct 31 03:06:18.634705 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.634655    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9dj5j" podStartSLOduration=5.746101621 podStartE2EDuration="11.634639554s" totalImagesPullingTime="5.888537933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:16.61461951 +0000 UTC m=+9.758726928" watchObservedRunningTime="2026-10-31 03:06:18.634639554 +0000 UTC m=+11.778746973"
Oct 31 03:06:18.743589 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.743563    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:06:18.744290 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.744240    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r8762" podStartSLOduration=3.408832213 podStartE2EDuration="11.744224101s" totalImagesPullingTime="8.335391888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:18.744110711 +0000 UTC m=+11.888218130" watchObservedRunningTime="2026-10-31 03:06:18.744224101 +0000 UTC m=+11.888331520"
Oct 31 03:06:18.849546 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:18.849475    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mlv4n" podStartSLOduration=3.29966418 podStartE2EDuration="11.849455136s" totalImagesPullingTime="8.549790956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:18.848764782 +0000 UTC m=+11.992872201" watchObservedRunningTime="2026-10-31 03:06:18.849455136 +0000 UTC m=+11.993562554"
Oct 31 03:06:19.361953 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.361811    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:06:18.743583425Z","UUID":"5953fb66-655e-4f41-9bdb-ce40f55afd81","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:06:19.363414 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.363385    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:06:19.363558 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.363455    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:06:19.424413 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.424381    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:19.424614 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:19.424509    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:19.424614 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.424561    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:19.424999 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:19.424681    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:19.529485 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.529439    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd" event={"ID":"4f85eaf9-d372-4e12-8016-0ade17b5badc","Type":"ContainerStarted","Data":"03806df8899db923447ea50de136eb9e0a1bc11aa6a1fad12930b25ec7bcec73"}
Oct 31 03:06:19.531782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.531750    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="c7a8b8a4cd5bb8bbd40bc51afe3351070d9990beba0b9bc10296cae46a563607" exitCode=0
Oct 31 03:06:19.531942 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:19.531838    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"c7a8b8a4cd5bb8bbd40bc51afe3351070d9990beba0b9bc10296cae46a563607"}
Oct 31 03:06:20.116063 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.115902    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:20.116387 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.116369    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:20.423982 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.423892    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:20.424127 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:20.424034    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:20.534788 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.534752    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brjlb" event={"ID":"5fb14716-1ea1-40f5-9ac7-7476c9daa35b","Type":"ContainerStarted","Data":"41681ce6623bd613d14e53ec4da7ee9745a2b815103dea1a32106292e66a2707"}
Oct 31 03:06:20.536178 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.536150    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c72n4" event={"ID":"ff779f60-ee44-48e3-b1fa-777374b9788b","Type":"ContainerStarted","Data":"e9e27b0c06e5bb8d5e008d7ea5c7dc24c967eb80739ce383c59c31e868103d16"}
Oct 31 03:06:20.748610 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:20.748553    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-brjlb" podStartSLOduration=5.044986524 podStartE2EDuration="13.748515624s" totalImagesPullingTime="8.7035291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.640434342 +0000 UTC m=+13.784541760" watchObservedRunningTime="2026-10-31 03:06:20.748515624 +0000 UTC m=+13.892623042"
Oct 31 03:06:21.424598 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.423890    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:21.424598 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.424048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:21.424598 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:21.424023    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:21.424598 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:21.424158    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:21.540437 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.540403    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd" event={"ID":"4f85eaf9-d372-4e12-8016-0ade17b5badc","Type":"ContainerStarted","Data":"4328de2dce6b0d3d804f2d66ca0921d80638058b1023326e6f5e5ceca85fb009"}
Oct 31 03:06:21.542725 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.542697    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="ce11711ac92e52c9f45b7fc065be55ff20ab4924610fbfac10f82e937ff55d94" exitCode=0
Oct 31 03:06:21.542852 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.542791    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"ce11711ac92e52c9f45b7fc065be55ff20ab4924610fbfac10f82e937ff55d94"}
Oct 31 03:06:21.650796 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.650737    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd" podStartSLOduration=4.226847541 podStartE2EDuration="14.650722343s" totalImagesPullingTime="10.423874802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:21.650403706 +0000 UTC m=+14.794511127" watchObservedRunningTime="2026-10-31 03:06:21.650722343 +0000 UTC m=+14.794829761"
Oct 31 03:06:21.650937 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:21.650910    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c72n4" podStartSLOduration=4.235510599 podStartE2EDuration="14.650906044s" totalImagesPullingTime="10.415395445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.748432387 +0000 UTC m=+13.892539805" watchObservedRunningTime="2026-10-31 03:06:21.650906044 +0000 UTC m=+14.795013493"
Oct 31 03:06:22.423931 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:22.423896    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:22.424126 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:22.424027    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:23.424102 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:23.424021    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:23.424738 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:23.424154    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:23.424738 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:23.424200    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:23.424738 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:23.424320    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:23.550552 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:23.550495    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="f626fc7f352dce9dd4312d4324476612b3ffb39e3a21db29d858f90ea240cdfc" exitCode=0
Oct 31 03:06:23.550727 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:23.550563    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"f626fc7f352dce9dd4312d4324476612b3ffb39e3a21db29d858f90ea240cdfc"}
Oct 31 03:06:24.423898 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:24.423705    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:24.424095 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:24.424019    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:25.135118 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:25.135088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:25.135118 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:25.135132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:25.135624 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.135245    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:25.135624 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.135303    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.135283885 +0000 UTC m=+34.279391283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:25.135624 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.135246    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:25.135624 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.135380    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.135362314 +0000 UTC m=+34.279469724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:25.235800 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:25.235763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:25.235992 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.235938    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:25.235992 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.235964    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:25.235992 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.235978    2570 projected.go:196] Error preparing data for projected volume kube-api-access-5drl7 for pod openshift-network-diagnostics/network-check-target-2nrm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:25.236154 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.236029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7 podName:37a46e66-bcbb-4bc9-b57b-deb78bc76adf nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.236016339 +0000 UTC m=+34.380123739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5drl7" (UniqueName: "kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7") pod "network-check-target-2nrm8" (UID: "37a46e66-bcbb-4bc9-b57b-deb78bc76adf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:25.424604 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:25.424509    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:25.424604 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:25.424546    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:25.424789 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.424658    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:25.424834 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:25.424810    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:26.424367 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:26.424330    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:26.424758 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:26.424439    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:27.426579 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.426341    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:27.426924 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.426385    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:27.426924 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:27.426650    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:27.426924 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:27.426682    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:27.561110 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561085    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:06:27.561423 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561395    2570 generic.go:356] "Generic (PLEG): container finished" podID="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" containerID="b2f4ae029a234d930f213693458bdb19d9f5b09883bfa9938ab25d4bac4034a3" exitCode=1
Oct 31 03:06:27.561545 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561466    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"d3551709a78efffededb65b2e3c269f1fafe1c78f87a06214245f93f8dc32247"}
Oct 31 03:06:27.561598 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561549    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"9fc7ea228d00adfee7f2598b9fd64c4a2f582b91acecb30d4a1bf59a135ebcb3"}
Oct 31 03:06:27.561642 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561601    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"dc2a442f374b0106a2d3fb6fee078e3d85a1a70e7f72dcce2e34cc50b8939991"}
Oct 31 03:06:27.561642 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561635    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"3cbd8bcad8f2672475db3b2e2d128e9b7dee4e70d0bda8eb8eb50bc933c250c3"}
Oct 31 03:06:27.561718 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561669    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerDied","Data":"b2f4ae029a234d930f213693458bdb19d9f5b09883bfa9938ab25d4bac4034a3"}
Oct 31 03:06:27.561756 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.561715    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"eafa71ce96dfa4dd219630993a04f4a15bf25122063f47463a5723811f1abdcb"}
Oct 31 03:06:27.566958 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.566936    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="d2a8053a9f87a3d980733b15ed56c8f03256e386463ec8f0561ebced3abb5d9e" exitCode=0
Oct 31 03:06:27.567060 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:27.566975    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"d2a8053a9f87a3d980733b15ed56c8f03256e386463ec8f0561ebced3abb5d9e"}
Oct 31 03:06:28.423459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:28.423428    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:28.423625 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:28.423554    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:28.572353 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:28.572318    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="fcff57fdf4eb85e850aa82d52b26c66ca97472503954692e0dcab5bb11b97abe" exitCode=0
Oct 31 03:06:28.572782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:28.572361    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerDied","Data":"fcff57fdf4eb85e850aa82d52b26c66ca97472503954692e0dcab5bb11b97abe"}
Oct 31 03:06:29.423694 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:29.423662    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:29.423854 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:29.423720    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:29.423906 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:29.423849    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:29.423973 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:29.423954    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:29.577552 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:29.577468    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:06:29.578063 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:29.577865    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"a3e5ff12dd6cfdc8512e4c1880fc0d4fb6feb2d7a5384f3ad4e2295f84852092"}
Oct 31 03:06:29.580917 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:29.580875    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" event={"ID":"dc279cb7-148d-4787-983c-ea212b73a3cc","Type":"ContainerStarted","Data":"b138ee0e58e98a211d89f82facfd6a0fba9b71e882383858eb26d4b7d65e93b9"}
Oct 31 03:06:30.424212 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:30.424178    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:30.424653 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:30.424626    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:31.424491 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.424465    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:31.425119 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:31.424671    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:31.425119 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.424465    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:31.425119 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:31.424801    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:31.588692 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.588668    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:06:31.589122 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.589100    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"a85fc0b02e53af17258993f09dd71f9209bdbf21e72a67bc91fb69ef1e7d52e6"}
Oct 31 03:06:31.589386 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.589365    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:31.589510 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.589437    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:31.589510 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.589453    2570 scope.go:122] "RemoveContainer" containerID="b2f4ae029a234d930f213693458bdb19d9f5b09883bfa9938ab25d4bac4034a3"
Oct 31 03:06:31.605028 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.605005    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:31.860008 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:31.859959    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m9b2j" podStartSLOduration=10.577898698 podStartE2EDuration="24.85994613s" totalImagesPullingTime="14.282047432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:29.747458691 +0000 UTC m=+22.891566121" watchObservedRunningTime="2026-10-31 03:06:31.85994613 +0000 UTC m=+25.004053547"
Oct 31 03:06:32.423650 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.423615    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:32.423819 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:32.423733    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:32.594639 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.594614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:06:32.595249 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.595000    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" event={"ID":"f8dcdbcd-d8e5-4400-880b-c89f3362af1b","Type":"ContainerStarted","Data":"43743ed337bb960bcad44120570e62d03407dcfa277eb712941a0ea51b398600"}
Oct 31 03:06:32.595249 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.595151    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:32.610356 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.610330    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:06:32.799707 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:32.799657    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd" podStartSLOduration=8.800079036 podStartE2EDuration="25.799643972s" totalImagesPullingTime="16.999564936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:32.799548171 +0000 UTC m=+25.943655579" watchObservedRunningTime="2026-10-31 03:06:32.799643972 +0000 UTC m=+25.943751389"
Oct 31 03:06:33.424363 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:33.424330    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:33.424576 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:33.424337    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:33.424576 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:33.424470    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:33.424576 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:33.424496    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:34.161445 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.161410    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bj5qd"]
Oct 31 03:06:34.161934 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.161560    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:34.161934 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:34.161662    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:34.161934 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.161924    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2nrm8"]
Oct 31 03:06:34.162081 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.162002    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:34.162081 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:34.162066    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:34.164429 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.164409    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5fjf"]
Oct 31 03:06:34.164523 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:34.164483    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:34.164601 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:34.164584    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:35.423653 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:35.423617    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:35.424065 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:35.423658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:35.424065 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:35.423779    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:35.424065 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:35.423881    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:35.892405 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:35.892231    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:35.892566 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:35.892507    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:06:35.892828 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:35.892810    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:06:36.424193 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:36.424160    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:36.424612 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:36.424259    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:37.424943 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:37.424907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:37.425365 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:37.425059    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:37.425365 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:37.425111    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:37.425365 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:37.425182    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:38.424007 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:38.423981    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:38.424162 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:38.424078    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nrm8" podUID="37a46e66-bcbb-4bc9-b57b-deb78bc76adf"
Oct 31 03:06:39.424274 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:39.424239    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:39.424696 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:39.424239    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:39.424696 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:39.424382    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5fjf" podUID="b8f2b00f-643e-412d-9ecb-94f0fd75cf29"
Oct 31 03:06:39.424696 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:39.424419    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj5qd" podUID="5feec31a-0c22-428a-9d9a-3c16b3cd6097"
Oct 31 03:06:40.205671 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.205643    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeReady"
Oct 31 03:06:40.205837 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.205813    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:06:40.320339 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.320310    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qqbzc"]
Oct 31 03:06:40.323861 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.323847    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:06:40.361125 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.361095    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jjkm4"]
Oct 31 03:06:40.364615 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.364601    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"]
Oct 31 03:06:40.364741 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.364724    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:06:40.367706 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.367689    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-675lv"]
Oct 31 03:06:40.367819 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.367803    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.370824 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.370805    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"]
Oct 31 03:06:40.370923 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.370910    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:06:40.374038 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.374018    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c645688cb-cddsx"]
Oct 31 03:06:40.374134 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.374109    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.377040 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.377023    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"]
Oct 31 03:06:40.377159 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.377144    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.381019 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.381004    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ldnrv"]
Oct 31 03:06:40.381126 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.381110    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.384178 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.384160    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tbchd"]
Oct 31 03:06:40.384259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.384232    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.386951 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.386935    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"]
Oct 31 03:06:40.387064 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.387048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.389838 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.389823    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"]
Oct 31 03:06:40.390210 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.389942    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:06:40.392728 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.392710    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.411546 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.411504    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6489c4dbb7-6rbbc"]
Oct 31 03:06:40.418106 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.418084    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"]
Oct 31 03:06:40.418780 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.418757    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.419647 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.419627    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.419747 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.419644    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.424158 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.424140    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 03:06:40.424287 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.424258    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nhzcr\""
Oct 31 03:06:40.426633 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.426615    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 03:06:40.428651 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.428632    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ksv4j"]
Oct 31 03:06:40.429027 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.429009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.433835 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.433817    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"]
Oct 31 03:06:40.433947 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.433937    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.436718 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.436700    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b48949c57-xszl6"]
Oct 31 03:06:40.436818 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.436809    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.439721 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.439704    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"]
Oct 31 03:06:40.439811 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.439797    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.439896 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.439877    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:40.442782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442766    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"]
Oct 31 03:06:40.442856 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442792    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qqbzc"]
Oct 31 03:06:40.442856 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442806    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jjkm4"]
Oct 31 03:06:40.442856 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442819    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-675lv"]
Oct 31 03:06:40.442856 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442832    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"]
Oct 31 03:06:40.442856 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442856    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"]
Oct 31 03:06:40.443030 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442875    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tbchd"]
Oct 31 03:06:40.443030 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442891    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:06:40.443030 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442901    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"]
Oct 31 03:06:40.443030 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.442919    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ldnrv"]
Oct 31 03:06:40.447431 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447410    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.447520 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447445    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.447520 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447485    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.447667 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447646    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.447719 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrhv\" (UniqueName: \"kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv\") pod \"kas-connection-checker-685895c7c9-qqbzc\" (UID: \"ead07c5f-c2fb-42de-9254-059fab1e0ab7\") " pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:06:40.447777 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447719    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.447777 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447756    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.447863 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5hk\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.447863 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.447863 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.447985 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.448016 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.447981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pd4\" (UniqueName: \"kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4\") pod \"kas-connection-checker-685895c7c9-675lv\" (UID: \"bfe907fb-0ddb-48d8-8a02-8ee398870770\") " pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:06:40.448067 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.448067 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.448152 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448129    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4jz\" (UniqueName: \"kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.448190 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxt8k\" (UniqueName: \"kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.448243 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448226    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.448284 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5596k\" (UniqueName: \"kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k\") pod \"kas-connection-checker-685895c7c9-jjkm4\" (UID: \"71bad7a7-3b59-4e50-96d9-59dcd2ec790c\") " pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:06:40.448314 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.448375 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.448360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.460623 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.460603    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 03:06:40.460623 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.460617    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 03:06:40.461343 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.461327    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6489c4dbb7-6rbbc"]
Oct 31 03:06:40.467655 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.467630    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"]
Oct 31 03:06:40.470511 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.470493    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"]
Oct 31 03:06:40.474269 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.474097    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c645688cb-cddsx"]
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480581    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480678    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480741    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480842    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480889    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480913    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480941    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.480961    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481011    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481039    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481040    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481064    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481058    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481145    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481150    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481160    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481224    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.481461 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.481334    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.530674 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.530614    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 03:06:40.535241 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.535216    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 03:06:40.536018 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.535997    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 03:06:40.536084 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.535997    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 03:06:40.549004 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.548979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.549105 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.549105 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549030    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtrq\" (UniqueName: \"kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.549105 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549051    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.549215 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549106    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.549215 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549151    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.549298 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.549342 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.549416 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549396    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.549452 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.549550 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.549598 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549572    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zps7s\" (UniqueName: \"kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.549644 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549602    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4jz\" (UniqueName: \"kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.549708 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtm42\" (UniqueName: \"kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42\") pod \"volume-data-source-validator-58c9f5f46d-xc87v\" (UID: \"dc8ddf63-307d-4908-bff6-df337941dfaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:06:40.549749 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549725    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xxs\" (UniqueName: \"kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.549782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.549836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549781    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.549836 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4cb\" (UniqueName: \"kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.549915 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.549864    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:40.549915 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.549997 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.549949    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.049934267 +0000 UTC m=+34.194041677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:40.550042 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.549995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.550042 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550138 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550138 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550187    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.550192    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.550204    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:40.550227 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550212    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5hk\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.550244    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.050228153 +0000 UTC m=+34.194335560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550347    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.550379    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550408    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550459 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.550422    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.050407785 +0000 UTC m=+34.194515187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550496    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550618    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550662    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pd4\" (UniqueName: \"kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4\") pod \"kas-connection-checker-685895c7c9-675lv\" (UID: \"bfe907fb-0ddb-48d8-8a02-8ee398870770\") " pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550753    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550783    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxt8k\" (UniqueName: \"kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550852    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.550901 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550881    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzj8\" (UniqueName: \"kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.550989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5596k\" (UniqueName: \"kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k\") pod \"kas-connection-checker-685895c7c9-jjkm4\" (UID: \"71bad7a7-3b59-4e50-96d9-59dcd2ec790c\") " pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxz8t\" (UniqueName: \"kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551099    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551196    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxk4t\" (UniqueName: \"kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t\") pod \"network-check-source-6f78d59f99-2zqjk\" (UID: \"27fe767b-0eac-42e2-a937-e13497bad6bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551265    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551282    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmwv\" (UniqueName: \"kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551368    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.551438 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551427    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.551955 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551461    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.551955 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.551955 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrhv\" (UniqueName: \"kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv\") pod \"kas-connection-checker-685895c7c9-qqbzc\" (UID: \"ead07c5f-c2fb-42de-9254-059fab1e0ab7\") " pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:06:40.551955 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551659    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.551955 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.551885    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.554401 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.554383    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.560877 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.560857    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 03:06:40.562307 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.562291    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b48949c57-xszl6"]
Oct 31 03:06:40.562904 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.562889    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"]
Oct 31 03:06:40.563439 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.563430    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ksv4j"]
Oct 31 03:06:40.564008 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.563992    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"]
Oct 31 03:06:40.566724 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.566702    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 03:06:40.566826 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.566742    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 03:06:40.566826 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.566742    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:06:40.566826 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.566773    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 03:06:40.566826 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.566788    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 03:06:40.569094 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569081    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:06:40.569151 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569111    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:06:40.569196 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569180    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xtr75\""
Oct 31 03:06:40.569241 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569206    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:06:40.569241 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569212    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 03:06:40.569241 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569230    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 03:06:40.569359 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569281    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 03:06:40.569359 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569288    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 03:06:40.569359 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.569309    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:06:40.572672 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.572651    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.573679 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.573662    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581007    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581016    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581006    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581048    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581063    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:40.581091 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581087    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 03:06:40.581348 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581099    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.581348 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581088    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.581348 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581131    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.581348 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.581205    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:06:40.608030 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.608004    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.608351 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.608332    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4jz\" (UniqueName: \"kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:40.608450 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.608390    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5hk\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:40.608515 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.608486    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxt8k\" (UniqueName: \"kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.608732 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.608716    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrhv\" (UniqueName: \"kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv\") pod \"kas-connection-checker-685895c7c9-qqbzc\" (UID: \"ead07c5f-c2fb-42de-9254-059fab1e0ab7\") " pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:06:40.609102 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.609082    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pd4\" (UniqueName: \"kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4\") pod \"kas-connection-checker-685895c7c9-675lv\" (UID: \"bfe907fb-0ddb-48d8-8a02-8ee398870770\") " pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:06:40.609256 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.609243    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5596k\" (UniqueName: \"kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k\") pod \"kas-connection-checker-685895c7c9-jjkm4\" (UID: \"71bad7a7-3b59-4e50-96d9-59dcd2ec790c\") " pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:06:40.635509 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.635485    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:06:40.652845 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.652824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.652937 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.652854    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.652937 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.652873    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.652937 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.652891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.653056 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.653112 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.653165 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.653165 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.653255 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653173    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.653255 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.653255 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.653387 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653257    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.653387 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653281    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.653387 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653293    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.653387 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzj8\" (UniqueName: \"kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.653590 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653565    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653873    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxz8t\" (UniqueName: \"kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653906    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653930    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653976    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxk4t\" (UniqueName: \"kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t\") pod \"network-check-source-6f78d59f99-2zqjk\" (UID: \"27fe767b-0eac-42e2-a937-e13497bad6bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.653980    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.653966 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmwv\" (UniqueName: \"kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654058    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654315    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654345    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654373    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtrq\" (UniqueName: \"kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.654396    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.15437607 +0000 UTC m=+34.298483492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:40.654448 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zps7s\" (UniqueName: \"kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.654862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.654646    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtm42\" (UniqueName: \"kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42\") pod \"volume-data-source-validator-58c9f5f46d-xc87v\" (UID: \"dc8ddf63-307d-4908-bff6-df337941dfaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:06:40.655202 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.655094    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:40.655202 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.655110    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:40.655202 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.655156    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.155141499 +0000 UTC m=+34.299248895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:40.655458 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.655426    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.656044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.655931    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.656141 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656096    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.656208 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656154    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.656208 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656194    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.656308 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656270    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656735    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xxs\" (UniqueName: \"kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656831    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656849    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4cb\" (UniqueName: \"kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656892    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.656911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.657080    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657196    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657256    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.157241015 +0000 UTC m=+34.301348415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.657303    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657314    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657351    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.157339794 +0000 UTC m=+34.301447194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657394    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.657400    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.657440 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:40.657450    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:41.157434157 +0000 UTC m=+34.301541569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:40.658105 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.658081    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.658916 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.658896    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.659078 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.659060    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.659373 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.659342    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.676196 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.676167    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:06:40.689923 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.689902    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:06:40.719987 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.719728    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:40.729614 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.729562    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxk4t\" (UniqueName: \"kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t\") pod \"network-check-source-6f78d59f99-2zqjk\" (UID: \"27fe767b-0eac-42e2-a937-e13497bad6bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:06:40.729734 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.729715    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zps7s\" (UniqueName: \"kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:40.730262 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.730148    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xxs\" (UniqueName: \"kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:40.730262 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.730214    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxz8t\" (UniqueName: \"kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.731464 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.730897    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.731464 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.731004    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtrq\" (UniqueName: \"kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.731464 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.731386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzj8\" (UniqueName: \"kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.731464 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.731433    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtm42\" (UniqueName: \"kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42\") pod \"volume-data-source-validator-58c9f5f46d-xc87v\" (UID: \"dc8ddf63-307d-4908-bff6-df337941dfaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:06:40.731735 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.731716    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmwv\" (UniqueName: \"kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:40.732423 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.732359    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4cb\" (UniqueName: \"kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:40.732523 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.732506    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:40.736706 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.736688    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:06:40.757750 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.757647    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:06:40.762975 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.762208    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:06:40.770848 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.770438    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:06:40.791941 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:40.791903    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:06:41.018122 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.018096    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qqbzc"]
Oct 31 03:06:41.018851 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.018826    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead07c5f_c2fb_42de_9254_059fab1e0ab7.slice/crio-0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7 WatchSource:0}: Error finding container 0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7: Status 404 returned error can't find the container with id 0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7
Oct 31 03:06:41.026883 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.026864    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:06:41.061652 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.061564    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:41.061652 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.061617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:41.061828 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.061808    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:41.061960 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.061944    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:41.062073 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062012    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.061991466 +0000 UTC m=+35.206098877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:41.062188 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062093    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:41.062188 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062105    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:41.062188 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062138    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.0621274 +0000 UTC m=+35.206234815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:41.062386 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062195    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:41.062386 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.062229    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.062217507 +0000 UTC m=+35.206324909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:41.063821 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.063795    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jjkm4"]
Oct 31 03:06:41.064979 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.064928    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bad7a7_3b59_4e50_96d9_59dcd2ec790c.slice/crio-56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46 WatchSource:0}: Error finding container 56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46: Status 404 returned error can't find the container with id 56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46
Oct 31 03:06:41.140261 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.140194    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"]
Oct 31 03:06:41.141557 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.141514    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"]
Oct 31 03:06:41.143272 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.143246    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b558d27_fb04_4431_88a3_22d27045dacb.slice/crio-9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e WatchSource:0}: Error finding container 9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e: Status 404 returned error can't find the container with id 9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e
Oct 31 03:06:41.162892 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.162860    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:41.163039 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.162973    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:41.163039 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163024    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.163011159 +0000 UTC m=+35.307118554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:41.163039 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.162973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:41.163198 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163089    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.163069985 +0000 UTC m=+35.307177401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:41.163198 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.163141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:41.163198 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.163175    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:41.163352 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163281    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:41.163352 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.163301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:41.163352 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163335    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret podName:5feec31a-0c22-428a-9d9a-3c16b3cd6097 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:13.163320156 +0000 UTC m=+66.307427571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret") pod "global-pull-secret-syncer-bj5qd" (UID: "5feec31a-0c22-428a-9d9a-3c16b3cd6097") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163368    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163286    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163400    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.163404    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163414    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs podName:b8f2b00f-643e-412d-9ecb-94f0fd75cf29 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:13.163403645 +0000 UTC m=+66.307511055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs") pod "network-metrics-daemon-s5fjf" (UID: "b8f2b00f-643e-412d-9ecb-94f0fd75cf29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.163441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163455    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.163440563 +0000 UTC m=+35.307547994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:41.163499 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163466    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:41.163902 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163503    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:41.163902 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163513    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.163500354 +0000 UTC m=+35.307607763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:41.163902 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:41.163566    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:42.163556602 +0000 UTC m=+35.307664028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:41.167985 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.167953    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d0ddb1_2f0c_451f_9909_846980b61b9f.slice/crio-afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b WatchSource:0}: Error finding container afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b: Status 404 returned error can't find the container with id afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b
Oct 31 03:06:41.170433 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.170414    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ksv4j"]
Oct 31 03:06:41.179726 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.179669    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"]
Oct 31 03:06:41.186067 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.186020    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"]
Oct 31 03:06:41.193374 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.193168    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-675lv"]
Oct 31 03:06:41.195858 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.195834    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe907fb_0ddb_48d8_8a02_8ee398870770.slice/crio-e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146 WatchSource:0}: Error finding container e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146: Status 404 returned error can't find the container with id e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146
Oct 31 03:06:41.206505 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.206487    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"]
Oct 31 03:06:41.206982 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.206961    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fe767b_0eac_42e2_a937_e13497bad6bd.slice/crio-625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764 WatchSource:0}: Error finding container 625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764: Status 404 returned error can't find the container with id 625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764
Oct 31 03:06:41.264664 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.264485    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:41.267494 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.267427    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:41.388260 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.388226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:41.428018 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.427990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:06:41.428018 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.428009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:06:41.434002 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.433974    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde8fbc1_b8e0_49a5_9725_4586869dce6c.slice/crio-f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff WatchSource:0}: Error finding container f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff: Status 404 returned error can't find the container with id f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff
Oct 31 03:06:41.434113 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.434057    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ldnrv"]
Oct 31 03:06:41.480253 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.480222    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:06:41.480391 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.480318    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:06:41.480391 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.480326    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:06:41.480391 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.480320    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:06:41.616704 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.616612    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ldnrv" event={"ID":"fde8fbc1-b8e0-49a5-9725-4586869dce6c","Type":"ContainerStarted","Data":"f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff"}
Oct 31 03:06:41.617761 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.617737    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286" event={"ID":"da80e5b0-536a-46a8-aba6-c89002fa216d","Type":"ContainerStarted","Data":"31a110bcda188a202bbc102cf068e93f628b4a13a46db8fb1864e23aff1cb41e"}
Oct 31 03:06:41.619044 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.619021    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-675lv" event={"ID":"bfe907fb-0ddb-48d8-8a02-8ee398870770","Type":"ContainerStarted","Data":"38d1769549153e74f93e2e174e1f01a1bd5ee3c2dbacfc6c3c5290d2a2ee0a9f"}
Oct 31 03:06:41.619156 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.619068    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-675lv" event={"ID":"bfe907fb-0ddb-48d8-8a02-8ee398870770","Type":"ContainerStarted","Data":"e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146"}
Oct 31 03:06:41.620129 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.620110    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk" event={"ID":"27fe767b-0eac-42e2-a937-e13497bad6bd","Type":"ContainerStarted","Data":"625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764"}
Oct 31 03:06:41.621236 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.621216    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ksv4j" event={"ID":"88d0ddb1-2f0c-451f-9909-846980b61b9f","Type":"ContainerStarted","Data":"afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b"}
Oct 31 03:06:41.622473 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.622455    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qqbzc" event={"ID":"ead07c5f-c2fb-42de-9254-059fab1e0ab7","Type":"ContainerStarted","Data":"26d619a13fbb0c5220f39113235feed42ee1f3e86d5e2d5100698be52bb0fb8b"}
Oct 31 03:06:41.622579 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.622486    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qqbzc" event={"ID":"ead07c5f-c2fb-42de-9254-059fab1e0ab7","Type":"ContainerStarted","Data":"0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7"}
Oct 31 03:06:41.623566 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.623547    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v" event={"ID":"dc8ddf63-307d-4908-bff6-df337941dfaf","Type":"ContainerStarted","Data":"2cd4c0559b736a091506203218322055a7d05526cd14e51c68266db5db133594"}
Oct 31 03:06:41.624627 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.624599    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" event={"ID":"1b558d27-fb04-4431-88a3-22d27045dacb","Type":"ContainerStarted","Data":"9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e"}
Oct 31 03:06:41.626001 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.625985    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jjkm4" event={"ID":"71bad7a7-3b59-4e50-96d9-59dcd2ec790c","Type":"ContainerStarted","Data":"2cdc4ef257b83d392a1191db0d417fad240e99fbd17ed6f73e2177fda5061e69"}
Oct 31 03:06:41.626076 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.626014    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jjkm4" event={"ID":"71bad7a7-3b59-4e50-96d9-59dcd2ec790c","Type":"ContainerStarted","Data":"56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46"}
Oct 31 03:06:41.627007 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.626989    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn" event={"ID":"0381e7c6-4dc6-4a42-a0cf-276261adcb26","Type":"ContainerStarted","Data":"5e91450e567c87cc9e49a9454ab7ce248a7bfd0079c7e3b92f119d5a85cdc67b"}
Oct 31 03:06:41.756139 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:41.756109    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2nrm8"]
Oct 31 03:06:41.756752 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:41.756729    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a46e66_bcbb_4bc9_b57b_deb78bc76adf.slice/crio-9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a WatchSource:0}: Error finding container 9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a: Status 404 returned error can't find the container with id 9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a
Oct 31 03:06:42.070826 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.070788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.070837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.070858    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.070947    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.070952    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.070976    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:42.071005 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.071008    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.070992829 +0000 UTC m=+37.215100243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:42.071251 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.071029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.07101301 +0000 UTC m=+37.215120408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:42.071251 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.071037    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:42.071251 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.071099    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.071081595 +0000 UTC m=+37.215189008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:42.172216 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.172186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:42.172400 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.172224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:42.172400 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.172308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:42.172400 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172351    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:42.172400 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.172374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172414    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.172396093 +0000 UTC m=+37.316503506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172464    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.172483    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172492    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172501    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.172485016 +0000 UTC m=+37.316592429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172556    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172566    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172567    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.172555195 +0000 UTC m=+37.316662592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172582    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.172575315 +0000 UTC m=+37.316682712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:42.172613 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:42.172601    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:44.172593719 +0000 UTC m=+37.316701115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:42.601700 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.601338    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-675lv" podStartSLOduration=372.601318713 podStartE2EDuration="6m12.601318713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:42.489431793 +0000 UTC m=+35.633539202" watchObservedRunningTime="2026-10-31 03:06:42.601318713 +0000 UTC m=+35.745426123"
Oct 31 03:06:42.642161 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.642119    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2nrm8" event={"ID":"37a46e66-bcbb-4bc9-b57b-deb78bc76adf","Type":"ContainerStarted","Data":"9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a"}
Oct 31 03:06:42.718934 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:42.717677    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qqbzc" podStartSLOduration=372.717656218 podStartE2EDuration="6m12.717656218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:42.603441251 +0000 UTC m=+35.747548670" watchObservedRunningTime="2026-10-31 03:06:42.717656218 +0000 UTC m=+35.861763637"
Oct 31 03:06:44.094049 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.094004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:44.094523 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.094066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:44.094523 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.094227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:44.094523 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094428    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:44.094523 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094496    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.094477571 +0000 UTC m=+41.238584970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:44.094784 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094595    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:44.094784 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094608    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:44.094784 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094647    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.094633899 +0000 UTC m=+41.238741296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:44.094784 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094710    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:44.094784 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.094743    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.094733134 +0000 UTC m=+41.238840537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:44.195025 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.194983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:44.195219 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.195082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:44.195219 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.195142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:44.195219 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195160    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:44.195219 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.195205    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195240    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.195220232 +0000 UTC m=+41.339327631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.195269    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195283    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195296    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195336    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.195319117 +0000 UTC m=+41.339426527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195393    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:44.195430 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195407    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.195395702 +0000 UTC m=+41.339503111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:44.195787 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195446    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.195436336 +0000 UTC m=+41.339543751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:44.195787 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195464    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:44.195787 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:44.195498    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.195487428 +0000 UTC m=+41.339594841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:44.651575 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.651518    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v" event={"ID":"dc8ddf63-307d-4908-bff6-df337941dfaf","Type":"ContainerStarted","Data":"04219d794c387a0dc5dc09bb8a620dcc43d5d7258bf375a4f330e54d7e4aacc6"}
Oct 31 03:06:44.766122 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.766057    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v" podStartSLOduration=353.989725677 podStartE2EDuration="5m56.766042578s" totalImagesPullingTime="2.776316901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:44.764695359 +0000 UTC m=+37.908802778" watchObservedRunningTime="2026-10-31 03:06:44.766042578 +0000 UTC m=+37.910149995"
Oct 31 03:06:44.766974 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:44.766930    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jjkm4" podStartSLOduration=374.766920326 podStartE2EDuration="6m14.766920326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:42.718917798 +0000 UTC m=+35.863025218" watchObservedRunningTime="2026-10-31 03:06:44.766920326 +0000 UTC m=+37.911027741"
Oct 31 03:06:45.657670 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:45.657404    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286" event={"ID":"da80e5b0-536a-46a8-aba6-c89002fa216d","Type":"ContainerStarted","Data":"0ff2c4155e619cf5ecaeac92c50d2600f8d62dc7c0eb5ec44911e43fa91d2520"}
Oct 31 03:06:46.673435 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.673406    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/0.log"
Oct 31 03:06:46.674303 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.673451    2570 generic.go:356] "Generic (PLEG): container finished" podID="1b558d27-fb04-4431-88a3-22d27045dacb" containerID="e3af2017ac23d3a44b726721e7564e0faa05ad9bba676047edec1ecb65250cfb" exitCode=255
Oct 31 03:06:46.674303 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.673927    2570 scope.go:122] "RemoveContainer" containerID="e3af2017ac23d3a44b726721e7564e0faa05ad9bba676047edec1ecb65250cfb"
Oct 31 03:06:46.674766 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.674722    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" event={"ID":"1b558d27-fb04-4431-88a3-22d27045dacb","Type":"ContainerDied","Data":"e3af2017ac23d3a44b726721e7564e0faa05ad9bba676047edec1ecb65250cfb"}
Oct 31 03:06:46.683312 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.683009    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ldnrv" event={"ID":"fde8fbc1-b8e0-49a5-9725-4586869dce6c","Type":"ContainerStarted","Data":"115a18fb41cbd6bbc355739534cc4748d397d068ccc3f28728a0a11dd9781005"}
Oct 31 03:06:46.793189 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.792665    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286" podStartSLOduration=331.526922099 podStartE2EDuration="5m35.792644594s" totalImagesPullingTime="4.265722495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:45.771419674 +0000 UTC m=+38.915527093" watchObservedRunningTime="2026-10-31 03:06:46.792644594 +0000 UTC m=+39.936752012"
Oct 31 03:06:46.907351 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:46.907298    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-ldnrv" podStartSLOduration=354.343916941 podStartE2EDuration="5m58.90728291s" totalImagesPullingTime="4.563365969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:46.90676598 +0000 UTC m=+40.050873399" watchObservedRunningTime="2026-10-31 03:06:46.90728291 +0000 UTC m=+40.051390331"
Oct 31 03:06:47.689008 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.688905    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/1.log"
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.690021    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/0.log"
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.690060    2570 generic.go:356] "Generic (PLEG): container finished" podID="1b558d27-fb04-4431-88a3-22d27045dacb" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07" exitCode=255
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.690153    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" event={"ID":"1b558d27-fb04-4431-88a3-22d27045dacb","Type":"ContainerDied","Data":"02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"}
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.690227    2570 scope.go:122] "RemoveContainer" containerID="e3af2017ac23d3a44b726721e7564e0faa05ad9bba676047edec1ecb65250cfb"
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.690738    2570 scope.go:122] "RemoveContainer" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"
Oct 31 03:06:47.691259 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:47.690967    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tkdn9_openshift-console-operator(1b558d27-fb04-4431-88a3-22d27045dacb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podUID="1b558d27-fb04-4431-88a3-22d27045dacb"
Oct 31 03:06:47.694003 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.693947    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn" event={"ID":"0381e7c6-4dc6-4a42-a0cf-276261adcb26","Type":"ContainerStarted","Data":"c251e0c29bf74642b9e91df0a8891d4ffe10c30cf0f15cfa2d0f2f69486310b0"}
Oct 31 03:06:47.719017 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.718987    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"]
Oct 31 03:06:47.725307 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.725283    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:06:47.727585 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.727562    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"]
Oct 31 03:06:47.731635 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.731615    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:06:47.744508 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.744481    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49g4k\" (UniqueName: \"kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k\") pod \"migrator-66df5d496b-l4gzk\" (UID: \"b5bfd162-e42e-423a-907c-1ae68fa18ec7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:06:47.744666 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.744652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46bj\" (UniqueName: \"kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj\") pod \"migrator-66df5d496b-4tsxv\" (UID: \"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:06:47.795356 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.795318    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"]
Oct 31 03:06:47.795721 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.795687    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:47.795721 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.795697    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 03:06:47.795897 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.795876    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-77xsg\""
Oct 31 03:06:47.825135 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.825102    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"]
Oct 31 03:06:47.845893 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.845863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g4k\" (UniqueName: \"kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k\") pod \"migrator-66df5d496b-l4gzk\" (UID: \"b5bfd162-e42e-423a-907c-1ae68fa18ec7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:06:47.846078 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.845960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46bj\" (UniqueName: \"kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj\") pod \"migrator-66df5d496b-4tsxv\" (UID: \"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:06:47.862481 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.862451    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-svcxz"]
Oct 31 03:06:47.866420 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.866394    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:47.910778 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.910745    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g4k\" (UniqueName: \"kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k\") pod \"migrator-66df5d496b-l4gzk\" (UID: \"b5bfd162-e42e-423a-907c-1ae68fa18ec7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:06:47.913228 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.913200    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46bj\" (UniqueName: \"kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj\") pod \"migrator-66df5d496b-4tsxv\" (UID: \"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:06:47.920678 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.920649    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:06:47.920809 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.920721    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:06:47.920977 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.920960    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:06:47.926562 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.924808    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svcxz"]
Oct 31 03:06:47.933212 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.933150    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn" podStartSLOduration=330.791306072 podStartE2EDuration="5m36.933133595s" totalImagesPullingTime="6.141827523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:47.932388362 +0000 UTC m=+41.076495782" watchObservedRunningTime="2026-10-31 03:06:47.933133595 +0000 UTC m=+41.077241013"
Oct 31 03:06:47.946556 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.946473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:47.946681 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.946576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:47.946837 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.946804    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:47.946909 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.946851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:47.946909 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:47.946902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpgl\" (UniqueName: \"kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.049961 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.049706    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:06:48.050506 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050237    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050506 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050506 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050506 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpgl\" (UniqueName: \"kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050506 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050769 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050683    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.050769 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.050761    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.050858 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.050803    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.55078845 +0000 UTC m=+41.694895846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.050858 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.050839    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.050970 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.050861    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:48.550853573 +0000 UTC m=+41.694960968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.050970 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.050948    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.056989 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.056964    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:06:48.080875 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.080847    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:06:48.085675 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.085647    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.110087 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.110013    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpgl\" (UniqueName: \"kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.140120 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.139908    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:06:48.145157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.145113    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mq4kc\""
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152178    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152300    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.152503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.152675    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.152740    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.152719323 +0000 UTC m=+49.296826741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.152816    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.152828    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:48.153071 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.152863    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.152849935 +0000 UTC m=+49.296957345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:48.154110 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.153851    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:48.154110 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.153909    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.153893545 +0000 UTC m=+49.298000964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:48.255082 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.255082 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255054    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:48.255082 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:48.255333 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:48.255333 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.255333 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:48.255333 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255274    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.255333 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.255554 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.255344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:48.255554 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.255474    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:48.255554 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.255487    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:48.255693 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.255556    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.255520731 +0000 UTC m=+49.399628131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:48.256246 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.256224    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.256738    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.256832    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.256877    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.256862758 +0000 UTC m=+49.400970163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.256932    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.256962    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.256952677 +0000 UTC m=+49.401060080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.257010    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.257038    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.257029115 +0000 UTC m=+49.401136516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.257097    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.257087321 +0000 UTC m=+49.401194719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:48.257361 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.257249    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.315325 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.315289    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.400235 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.400213    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:06:48.448024 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.447992    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"]
Oct 31 03:06:48.449027 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:48.448994    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfbbf60_9b29_45f3_be5a_0f1282beb2e9.slice/crio-c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136 WatchSource:0}: Error finding container c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136: Status 404 returned error can't find the container with id c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136
Oct 31 03:06:48.494197 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.494150    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"]
Oct 31 03:06:48.497288 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:48.497258    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bfd162_e42e_423a_907c_1ae68fa18ec7.slice/crio-f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2 WatchSource:0}: Error finding container f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2: Status 404 returned error can't find the container with id f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2
Oct 31 03:06:48.558746 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.558657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.558914 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.558768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:48.558973 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.558938    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.559027 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.558986    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:49.558973733 +0000 UTC m=+42.703081132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.559027 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.559020    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.559127 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.559040    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:49.55903411 +0000 UTC m=+42.703141507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.698440 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.698411    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk" event={"ID":"27fe767b-0eac-42e2-a937-e13497bad6bd","Type":"ContainerStarted","Data":"10f0c8dbf1eaec2928769f21b0c3c46d60189bd08242c058de68a81191feeff8"}
Oct 31 03:06:48.700299 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.700278    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/1.log"
Oct 31 03:06:48.700646 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.700626    2570 scope.go:122] "RemoveContainer" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"
Oct 31 03:06:48.700853 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:48.700831    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tkdn9_openshift-console-operator(1b558d27-fb04-4431-88a3-22d27045dacb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podUID="1b558d27-fb04-4431-88a3-22d27045dacb"
Oct 31 03:06:48.701866 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.701843    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2nrm8" event={"ID":"37a46e66-bcbb-4bc9-b57b-deb78bc76adf","Type":"ContainerStarted","Data":"f5d9f57a490fabce7ad239f1db6c98305dda746c14371327f43dd7d7fe7c0202"}
Oct 31 03:06:48.701935 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.701914    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:06:48.703024 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.703000    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk" event={"ID":"b5bfd162-e42e-423a-907c-1ae68fa18ec7","Type":"ContainerStarted","Data":"f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2"}
Oct 31 03:06:48.704119 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.704096    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv" event={"ID":"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9","Type":"ContainerStarted","Data":"c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136"}
Oct 31 03:06:48.776506 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:06:48.776471    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb3c50f_ca36_437c_b720_034fa17cb3e1.slice/crio-92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5 WatchSource:0}: Error finding container 92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5: Status 404 returned error can't find the container with id 92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5
Oct 31 03:06:48.777339 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.777312    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:06:48.811497 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.811302    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk" podStartSLOduration=331.068955233 podStartE2EDuration="5m37.811284186s" totalImagesPullingTime="6.742328953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:48.810366966 +0000 UTC m=+41.954474383" watchObservedRunningTime="2026-10-31 03:06:48.811284186 +0000 UTC m=+41.955391604"
Oct 31 03:06:48.926066 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:48.926014    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2nrm8" podStartSLOduration=35.325956095 podStartE2EDuration="41.926000084s" totalImagesPullingTime="6.600043989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:48.925971594 +0000 UTC m=+42.070079012" watchObservedRunningTime="2026-10-31 03:06:48.926000084 +0000 UTC m=+42.070107499"
Oct 31 03:06:49.570458 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.570423    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:49.570685 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:49.570590    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:49.570685 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.570612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:49.570685 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:49.570670    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:51.570652446 +0000 UTC m=+44.714759862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:49.570841 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:49.570710    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:49.570841 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:49.570744    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:51.570733885 +0000 UTC m=+44.714841294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:49.709749 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.709700    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ksv4j" event={"ID":"88d0ddb1-2f0c-451f-9909-846980b61b9f","Type":"ContainerStarted","Data":"77e5f808372e4dad8ebfb6404efeba0d88aedda34512037d67183ef925488a75"}
Oct 31 03:06:49.711473 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.711436    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" event={"ID":"7cb3c50f-ca36-437c-b720-034fa17cb3e1","Type":"ContainerStarted","Data":"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"}
Oct 31 03:06:49.711592 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.711487    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" event={"ID":"7cb3c50f-ca36-437c-b720-034fa17cb3e1","Type":"ContainerStarted","Data":"92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5"}
Oct 31 03:06:49.939338 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:49.939288    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ksv4j" podStartSLOduration=2.452131823 podStartE2EDuration="9.939273772s" totalImagesPullingTime="7.487141949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:49.823429357 +0000 UTC m=+42.967536779" watchObservedRunningTime="2026-10-31 03:06:49.939273772 +0000 UTC m=+43.083381190"
Oct 31 03:06:50.716172 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.716133    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk" event={"ID":"b5bfd162-e42e-423a-907c-1ae68fa18ec7","Type":"ContainerStarted","Data":"59ef128d5a5edf1496047653b4a075b53a02fcb83523cb288c9d68b1d136f75f"}
Oct 31 03:06:50.717782 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.717748    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv" event={"ID":"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9","Type":"ContainerStarted","Data":"e0861a467c0b84d043d4d46e96b2d9707ecf5c7228e9d0096d56a2d8578beffb"}
Oct 31 03:06:50.720157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.720145    2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:50.720224 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.720169    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:06:50.720423 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.720410    2570 scope.go:122] "RemoveContainer" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"
Oct 31 03:06:50.720586 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:50.720565    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tkdn9_openshift-console-operator(1b558d27-fb04-4431-88a3-22d27045dacb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podUID="1b558d27-fb04-4431-88a3-22d27045dacb"
Oct 31 03:06:50.830742 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.830690    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" podStartSLOduration=2.8306765 podStartE2EDuration="2.8306765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:49.938434558 +0000 UTC m=+43.082541977" watchObservedRunningTime="2026-10-31 03:06:50.8306765 +0000 UTC m=+43.974783914"
Oct 31 03:06:50.831316 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.831292    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk" podStartSLOduration=1.785368745 podStartE2EDuration="3.831283475s" totalImagesPullingTime="2.04591473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:50.829959167 +0000 UTC m=+43.974066587" watchObservedRunningTime="2026-10-31 03:06:50.831283475 +0000 UTC m=+43.975390893"
Oct 31 03:06:50.945002 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:50.944955    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv" podStartSLOduration=1.9783981659999998 podStartE2EDuration="3.944940811s" totalImagesPullingTime="1.966542645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:50.944137683 +0000 UTC m=+44.088245100" watchObservedRunningTime="2026-10-31 03:06:50.944940811 +0000 UTC m=+44.089048228"
Oct 31 03:06:51.591588 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:51.591560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:51.591736 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:51.591653    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:51.591832 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:51.591812    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:51.591925 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:51.591911    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:55.591889002 +0000 UTC m=+48.735996419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:51.592190 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:51.592170    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:52.862834 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:52.862805    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mlv4n_d9b1e5a2-6307-4472-9502-5e97c55ac76f/dns-node-resolver/0.log"
Oct 31 03:06:53.401124 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:53.401101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9dj5j_801996f5-e5f7-4ef2-93b0-ce1b36c94343/node-ca/0.log"
Oct 31 03:06:54.200426 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:54.200393    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ksv4j_88d0ddb1-2f0c-451f-9909-846980b61b9f/serve-healthcheck-canary/0.log"
Oct 31 03:06:54.601033 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:54.600969    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4tsxv_ccfbbf60-9b29-45f3-be5a-0f1282beb2e9/migrator/0.log"
Oct 31 03:06:54.801470 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:54.801445    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-l4gzk_b5bfd162-e42e-423a-907c-1ae68fa18ec7/migrator/0.log"
Oct 31 03:06:55.000401 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:55.000376    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2d286_da80e5b0-536a-46a8-aba6-c89002fa216d/kube-storage-version-migrator-operator/0.log"
Oct 31 03:06:55.627959 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:55.627924    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:06:55.628373 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:55.628079    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:55.628373 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:55.628143    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:07:03.628128074 +0000 UTC m=+56.772235473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:56.233066 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.233028    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.233089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") pod \"image-registry-7c645688cb-cddsx\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") " pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.233111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233201    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233204    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233237    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233252    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls podName:009cba45-9f0a-4991-8ab1-86b7ed158f79 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.233239338 +0000 UTC m=+65.377346747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-26sqf" (UID: "009cba45-9f0a-4991-8ab1-86b7ed158f79") : secret "samples-operator-tls" not found
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233252    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c645688cb-cddsx: secret "image-registry-tls" not found
Oct 31 03:06:56.233268 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233263    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert podName:01f13667-aa82-4f8e-b860-abcfb602fc4c nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.233257676 +0000 UTC m=+65.377365072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dbpk4" (UID: "01f13667-aa82-4f8e-b860-abcfb602fc4c") : secret "networking-console-plugin-cert" not found
Oct 31 03:06:56.233556 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.233292    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls podName:864a9808-fe05-4b86-99e1-cd1f9ead161d nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.233280168 +0000 UTC m=+65.377387564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls") pod "image-registry-7c645688cb-cddsx" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d") : secret "image-registry-tls" not found
Oct 31 03:06:56.334502 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.334468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:56.334672 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.334522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:06:56.334672 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.334577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:06:56.334672 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.334593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:06:56.334672 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334646    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.334626315 +0000 UTC m=+65.478733736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : configmap references non-existent config key: service-ca.crt
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334690    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334695    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334705    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: secret "image-registry-tls" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334712    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334733    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls podName:c7cf6d1f-3302-4a26-85c1-5a6a8a06f972 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.334723955 +0000 UTC m=+65.478831351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls") pod "dns-default-tbchd" (UID: "c7cf6d1f-3302-4a26-85c1-5a6a8a06f972") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334745    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.334739453 +0000 UTC m=+65.478846850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : secret "image-registry-tls" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334767    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs podName:d608bf35-2830-4486-a05f-ac59ddb3c8e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.334749782 +0000 UTC m=+65.478857196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs") pod "router-default-6489c4dbb7-6rbbc" (UID: "d608bf35-2830-4486-a05f-ac59ddb3c8e6") : secret "router-metrics-certs-default" not found
Oct 31 03:06:56.334892 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:56.334792    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:06:56.335120 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334907    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:56.335120 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:56.334937    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls podName:ec33cd8c-1d9c-437b-8742-661d57128951 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:12.334928865 +0000 UTC m=+65.479036288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xv98j" (UID: "ec33cd8c-1d9c-437b-8742-661d57128951") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:06:59.424365 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.424334    2570 scope.go:122] "RemoveContainer" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"
Oct 31 03:06:59.747255 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.747231    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:06:59.747653 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.747636    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/1.log"
Oct 31 03:06:59.747710 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.747672    2570 generic.go:356] "Generic (PLEG): container finished" podID="1b558d27-fb04-4431-88a3-22d27045dacb" containerID="58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010" exitCode=255
Oct 31 03:06:59.747765 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.747748    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" event={"ID":"1b558d27-fb04-4431-88a3-22d27045dacb","Type":"ContainerDied","Data":"58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010"}
Oct 31 03:06:59.747814 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.747803    2570 scope.go:122] "RemoveContainer" containerID="02b26962b0222f1e10974b4ec4b087ba061a9b0a155e2ac62ba5eb4fb2268b07"
Oct 31 03:06:59.748121 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:06:59.748094    2570 scope.go:122] "RemoveContainer" containerID="58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010"
Oct 31 03:06:59.748332 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:06:59.748302    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tkdn9_openshift-console-operator(1b558d27-fb04-4431-88a3-22d27045dacb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podUID="1b558d27-fb04-4431-88a3-22d27045dacb"
Oct 31 03:07:00.720932 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:00.720891    2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:00.721336 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:00.720953    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:00.752165 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:00.752142    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:07:00.752460 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:00.752444    2570 scope.go:122] "RemoveContainer" containerID="58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010"
Oct 31 03:07:00.752659 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:07:00.752642    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tkdn9_openshift-console-operator(1b558d27-fb04-4431-88a3-22d27045dacb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podUID="1b558d27-fb04-4431-88a3-22d27045dacb"
Oct 31 03:07:03.703798 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:03.703768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:03.704173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:07:03.703943    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:07:03.704173 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:07:03.704023    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls podName:219ae079-d5b4-4aed-bf97-4766b3a2325d nodeName:}" failed. No retries permitted until 2026-10-31 03:07:19.704006559 +0000 UTC m=+72.848113955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svcxz" (UID: "219ae079-d5b4-4aed-bf97-4766b3a2325d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:07:04.614127 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:04.614101    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:07.768692 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.768657    2570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c645688cb-cddsx"]
Oct 31 03:07:07.769085 ip-10-0-140-194 kubenswrapper[2570]: E1031 03:07:07.768851    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7c645688cb-cddsx" podUID="864a9808-fe05-4b86-99e1-cd1f9ead161d"
Oct 31 03:07:07.774930 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.774907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:07:07.781165 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.781145    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:07:07.943896 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.943862    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944061 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.943911    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944061 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.943967    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944061 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.943991    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944061 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944050    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944283 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944082    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944283 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944213    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\") pod \"864a9808-fe05-4b86-99e1-cd1f9ead161d\" (UID: \"864a9808-fe05-4b86-99e1-cd1f9ead161d\") "
Oct 31 03:07:07.944390 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944309    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:07:07.944635 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944614    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/864a9808-fe05-4b86-99e1-cd1f9ead161d-ca-trust-extracted\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:07.944799 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944775    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:07.944862 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.944785    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:07.947702 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.947674    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:07.947801 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.947696    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "kube-api-access-ks5hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:07.947801 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.947749    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:07:07.947872 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:07.947796    2570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets" pod "864a9808-fe05-4b86-99e1-cd1f9ead161d" (UID: "864a9808-fe05-4b86-99e1-cd1f9ead161d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045169    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-installation-pull-secrets\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045197    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-trusted-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045207    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-certificates\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045218    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5hk\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-kube-api-access-ks5hk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045228    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-bound-sa-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.045252 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.045238    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/864a9808-fe05-4b86-99e1-cd1f9ead161d-image-registry-private-configuration\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:08.778010 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:08.777981    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c645688cb-cddsx"
Oct 31 03:07:09.030275 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:09.030200    2570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c645688cb-cddsx"]
Oct 31 03:07:09.033521 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:09.033502    2570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c645688cb-cddsx"]
Oct 31 03:07:09.156131 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:09.156101    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/864a9808-fe05-4b86-99e1-cd1f9ead161d-registry-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:09.430960 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:09.430911    2570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864a9808-fe05-4b86-99e1-cd1f9ead161d" path="/var/lib/kubelet/pods/864a9808-fe05-4b86-99e1-cd1f9ead161d/volumes"
Oct 31 03:07:12.285759 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.285706    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:12.286188 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.285804    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:12.288273 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.288246    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:12.288378 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.288308    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:12.387085 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:12.387268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:12.387268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:12.387268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387176    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:12.387268 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387223    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:12.387747 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.387724    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:12.389650 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.389628    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:12.389819 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.389799    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:12.390099 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.390081    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:12.390222 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.390202    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:12.497900 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.497871    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:12.535075 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.535041    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nhzcr\""
Oct 31 03:07:12.543193 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.543114    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:12.544183 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.544159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:12.548956 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.548000    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:12.584950 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.584912    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:07:12.595741 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.595353    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:12.632138 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.632080    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xtr75\""
Oct 31 03:07:12.633167 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.633135    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:12.870058 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.870029    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"]
Oct 31 03:07:12.975265 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.975243    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"]
Oct 31 03:07:12.976157 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:12.976135    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec33cd8c_1d9c_437b_8742_661d57128951.slice/crio-1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d WatchSource:0}: Error finding container 1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d: Status 404 returned error can't find the container with id 1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d
Oct 31 03:07:12.984187 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:12.984164    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f13667_aa82_4f8e_b860_abcfb602fc4c.slice/crio-51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b WatchSource:0}: Error finding container 51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b: Status 404 returned error can't find the container with id 51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b
Oct 31 03:07:12.985241 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:12.985220    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"]
Oct 31 03:07:13.025330 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:13.025300    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd608bf35_2830_4486_a05f_ac59ddb3c8e6.slice/crio-bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24 WatchSource:0}: Error finding container bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24: Status 404 returned error can't find the container with id bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24
Oct 31 03:07:13.025654 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.025629    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tbchd"]
Oct 31 03:07:13.026357 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:13.026333    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cf6d1f_3302_4a26_85c1_5a6a8a06f972.slice/crio-cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126 WatchSource:0}: Error finding container cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126: Status 404 returned error can't find the container with id cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126
Oct 31 03:07:13.027287 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.027270    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6489c4dbb7-6rbbc"]
Oct 31 03:07:13.029307 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:13.029290    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b4593e_3511_4f4c_8dc6_b2599c75efe4.slice/crio-cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e WatchSource:0}: Error finding container cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e: Status 404 returned error can't find the container with id cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e
Oct 31 03:07:13.034095 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.034077    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b48949c57-xszl6"]
Oct 31 03:07:13.195275 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.195235    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:13.195275 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.195273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:13.246704 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.246675    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:13.246858 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.246717    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:13.257864 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.257835    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:13.257993 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.257890    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:13.597497 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.597419    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:13.605929 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.605910    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:13.606158 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.606143    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:13.615308 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.615286    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:13.798335 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.798294    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf" event={"ID":"009cba45-9f0a-4991-8ab1-86b7ed158f79","Type":"ContainerStarted","Data":"557f33a737c3bc45193294dbc9136f5e6058eb9a041b5b2c5c84f43318ed5edc"}
Oct 31 03:07:13.799899 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.799858    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" event={"ID":"54b4593e-3511-4f4c-8dc6-b2599c75efe4","Type":"ContainerStarted","Data":"14160c639162f339c0c937614dcf097f245d208d8747b47bd4292a3772b4e6a4"}
Oct 31 03:07:13.800012 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.799908    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" event={"ID":"54b4593e-3511-4f4c-8dc6-b2599c75efe4","Type":"ContainerStarted","Data":"cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e"}
Oct 31 03:07:13.800012 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.799987    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:13.801057 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.801030    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j" event={"ID":"ec33cd8c-1d9c-437b-8742-661d57128951","Type":"ContainerStarted","Data":"1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d"}
Oct 31 03:07:13.802079 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.802060    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbchd" event={"ID":"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972","Type":"ContainerStarted","Data":"cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126"}
Oct 31 03:07:13.803449 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.803428    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc" event={"ID":"d608bf35-2830-4486-a05f-ac59ddb3c8e6","Type":"ContainerStarted","Data":"e90ed73afd0c34dfc9584d22ca6a44ddf9ea8382c2e836f68905fda03a7decc5"}
Oct 31 03:07:13.803555 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.803467    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc" event={"ID":"d608bf35-2830-4486-a05f-ac59ddb3c8e6","Type":"ContainerStarted","Data":"bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24"}
Oct 31 03:07:13.804399 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:13.804382    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4" event={"ID":"01f13667-aa82-4f8e-b860-abcfb602fc4c","Type":"ContainerStarted","Data":"51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b"}
Oct 31 03:07:13.999847 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:13.999817    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feec31a_0c22_428a_9d9a_3c16b3cd6097.slice/crio-540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9 WatchSource:0}: Error finding container 540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9: Status 404 returned error can't find the container with id 540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9
Oct 31 03:07:14.000660 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.000634    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bj5qd"]
Oct 31 03:07:14.015327 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.015285    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" podStartSLOduration=57.015272834 podStartE2EDuration="57.015272834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:14.014947276 +0000 UTC m=+67.159054694" watchObservedRunningTime="2026-10-31 03:07:14.015272834 +0000 UTC m=+67.159380249"
Oct 31 03:07:14.036644 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.036620    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5fjf"]
Oct 31 03:07:14.045385 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:14.045358    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f2b00f_643e_412d_9ecb_94f0fd75cf29.slice/crio-38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a WatchSource:0}: Error finding container 38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a: Status 404 returned error can't find the container with id 38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a
Oct 31 03:07:14.134428 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.134377    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc" podStartSLOduration=363.134364058 podStartE2EDuration="6m3.134364058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:14.133552433 +0000 UTC m=+67.277659851" watchObservedRunningTime="2026-10-31 03:07:14.134364058 +0000 UTC m=+67.278471476"
Oct 31 03:07:14.549192 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.549155    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:14.551820 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.551795    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:14.812369 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.812275    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5fjf" event={"ID":"b8f2b00f-643e-412d-9ecb-94f0fd75cf29","Type":"ContainerStarted","Data":"38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a"}
Oct 31 03:07:14.813763 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.813735    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bj5qd" event={"ID":"5feec31a-0c22-428a-9d9a-3c16b3cd6097","Type":"ContainerStarted","Data":"540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9"}
Oct 31 03:07:14.814297 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.814274    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:14.815304 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:14.815284    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:16.823465 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:16.823428    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4" event={"ID":"01f13667-aa82-4f8e-b860-abcfb602fc4c","Type":"ContainerStarted","Data":"c13e4a14bdd36c456424791dcd9395f5608784d04986d18b0cf8c7a0e0181741"}
Oct 31 03:07:16.946462 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:16.946408    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4" podStartSLOduration=334.956011457 podStartE2EDuration="5m37.946393974s" totalImagesPullingTime="2.990382517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:16.944413798 +0000 UTC m=+70.088521216" watchObservedRunningTime="2026-10-31 03:07:16.946393974 +0000 UTC m=+70.090501398"
Oct 31 03:07:18.834181 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:18.834148    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j" event={"ID":"ec33cd8c-1d9c-437b-8742-661d57128951","Type":"ContainerStarted","Data":"f3f5030fca7e658bb72290cdeeb49fca01be932afc387255024599175cdafc50"}
Oct 31 03:07:18.948601 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:18.948506    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j" podStartSLOduration=362.944114137 podStartE2EDuration="6m7.948491857s" totalImagesPullingTime="5.00437772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:18.948467728 +0000 UTC m=+72.092575146" watchObservedRunningTime="2026-10-31 03:07:18.948491857 +0000 UTC m=+72.092599276"
Oct 31 03:07:19.715223 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.715190    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:07:19.756157 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.756117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:19.758776 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.758747    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:19.840810 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.840778    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbchd" event={"ID":"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972","Type":"ContainerStarted","Data":"41fdb66402ab632c68ad5058208d88acf81463eedabf2cb316f3eade5967e77a"}
Oct 31 03:07:19.841245 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.840831    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tbchd" event={"ID":"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972","Type":"ContainerStarted","Data":"c2c83d264d265e3901153d88cfee64f1766ddd20c8eec903629712da8f3b2e0d"}
Oct 31 03:07:19.841245 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.840880    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:19.842813 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.842785    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5fjf" event={"ID":"b8f2b00f-643e-412d-9ecb-94f0fd75cf29","Type":"ContainerStarted","Data":"fc54604e8dc31e8dc3d3a691afc3145869dffecd5495f2dc6dde9511937d2584"}
Oct 31 03:07:19.842944 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.842824    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5fjf" event={"ID":"b8f2b00f-643e-412d-9ecb-94f0fd75cf29","Type":"ContainerStarted","Data":"3227a7d924a0ce22ed0225cdf16a7ae609449e07e96410e377cf1bf576315a01"}
Oct 31 03:07:19.954865 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:19.954814    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-tbchd" podStartSLOduration=33.918286391 podStartE2EDuration="39.954799208s" totalImagesPullingTime="6.036512817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:19.954415164 +0000 UTC m=+73.098522583" watchObservedRunningTime="2026-10-31 03:07:19.954799208 +0000 UTC m=+73.098906650"
Oct 31 03:07:20.038034 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.037962    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:20.047071 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.047045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:20.070505 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.070447    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s5fjf" podStartSLOduration=68.38814332 podStartE2EDuration="1m13.07042754s" totalImagesPullingTime="4.68228422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:20.069595449 +0000 UTC m=+73.213702869" watchObservedRunningTime="2026-10-31 03:07:20.07042754 +0000 UTC m=+73.214534962"
Oct 31 03:07:20.469927 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.469896    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svcxz"]
Oct 31 03:07:20.471202 ip-10-0-140-194 kubenswrapper[2570]: W1031 03:07:20.471178    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219ae079_d5b4_4aed_bf97_4766b3a2325d.slice/crio-a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d WatchSource:0}: Error finding container a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d: Status 404 returned error can't find the container with id a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d
Oct 31 03:07:20.849414 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.849316    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svcxz" event={"ID":"219ae079-d5b4-4aed-bf97-4766b3a2325d","Type":"ContainerStarted","Data":"9c37d94748f374c45b322ef467003e24dc184584ca98d4bb0d3d4c5144acca4a"}
Oct 31 03:07:20.849414 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:20.849382    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svcxz" event={"ID":"219ae079-d5b4-4aed-bf97-4766b3a2325d","Type":"ContainerStarted","Data":"a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d"}
Oct 31 03:07:22.424739 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.424658    2570 scope.go:122] "RemoveContainer" containerID="58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010"
Oct 31 03:07:22.859681 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.859656    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:07:22.859854 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.859745    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" event={"ID":"1b558d27-fb04-4431-88a3-22d27045dacb","Type":"ContainerStarted","Data":"783a2992f8d6dde39e5213a5d8910ef54911c28b14ba83b95405435ef7076bf2"}
Oct 31 03:07:22.860089 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.860037    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:22.861808 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.861781    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svcxz" event={"ID":"219ae079-d5b4-4aed-bf97-4766b3a2325d","Type":"ContainerStarted","Data":"8f131aa2e2d4c6b21f874c27f7a4539e14cafa360b30f9eae1150e30daa5175d"}
Oct 31 03:07:22.976286 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:22.976220    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9" podStartSLOduration=391.442045424 podStartE2EDuration="6m35.976202744s" totalImagesPullingTime="4.53415732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:22.975268637 +0000 UTC m=+76.119376055" watchObservedRunningTime="2026-10-31 03:07:22.976202744 +0000 UTC m=+76.120310353"
Oct 31 03:07:23.109837 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:23.109757    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:26.883009 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:26.882972    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svcxz" event={"ID":"219ae079-d5b4-4aed-bf97-4766b3a2325d","Type":"ContainerStarted","Data":"04df00bce9c1879261900fd18fcc5c0abdc5cc8d4569430cbf3de71371959acb"}
Oct 31 03:07:27.000121 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:27.000073    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-svcxz" podStartSLOduration=34.517506407 podStartE2EDuration="40.000060761s" totalImagesPullingTime="5.482554354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:26.998898168 +0000 UTC m=+80.143005586" watchObservedRunningTime="2026-10-31 03:07:27.000060761 +0000 UTC m=+80.144168198"
Oct 31 03:07:29.852130 ip-10-0-140-194 kubenswrapper[2570]: I1031 03:07:29.852056    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:29.894580 ip-10-0-140-194 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 03:07:29.917974 ip-10-0-140-194 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 03:07:29.918162 ip-10-0-140-194 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 03:07:29.918370 ip-10-0-140-194 systemd[1]: kubelet.service: Consumed 4.907s CPU time.
Oct 31 03:07:29.940181 ip-10-0-140-194 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:07:30.022681 ip-10-0-140-194 kubenswrapper[5831]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:30.022681 ip-10-0-140-194 kubenswrapper[5831]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:07:30.022681 ip-10-0-140-194 kubenswrapper[5831]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:30.022681 ip-10-0-140-194 kubenswrapper[5831]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:30.025591 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025463    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.025591 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025586    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.025591 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025591    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.025591 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025596    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.025591 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025600    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025605    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025609    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025613    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025616    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025620    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025624    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025628    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025634    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025640    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025656    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025662    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025666    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025670    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025674    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025678    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025681    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025686    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.025942 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025690    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025695    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025699    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025703    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025707    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025712    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025728    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025732    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025736    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025741    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025746    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025754    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025758    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025762    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025767    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025771    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025775    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025780    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025784    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.026743 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025788    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025792    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025797    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025801    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025805    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025809    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025815    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025820    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025825    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025830    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025834    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025839    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025843    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025847    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025852    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025858    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025862    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025866    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025870    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025875    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.027645 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025879    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025883    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025888    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025892    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025898    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025902    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025907    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025911    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025916    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025920    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025924    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025928    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025933    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025937    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025941    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025945    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025949    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025953    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025957    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025962    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.028516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025967    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025971    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025976    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025980    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025985    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025990    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025995    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.025999    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026003    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026007    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026011    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026016    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026021    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026025    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026029    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026034    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026039    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026044    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026048    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026054    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.029473 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026059    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026063    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026067    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026071    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.026076    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027348    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027359    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027365    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027370    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027374    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027379    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027383    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027388    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027393    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027398    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027402    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027406    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027411    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027416    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027420    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.030436 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027425    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027429    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027433    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027438    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027442    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027447    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027451    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027455    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027459    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027464    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027469    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027473    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027477    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027481    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027486    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027490    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027494    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027498    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027503    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.031365 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027507    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027511    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027516    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027566    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027574    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027582    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027589    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027595    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027599    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027603    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027607    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027613    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027618    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027622    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027626    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027631    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027635    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027639    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027643    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.032191 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027647    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027651    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027657    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027661    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027665    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027670    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027674    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027678    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027686    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027690    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027694    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027699    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027704    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027708    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027713    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027717    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027721    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027725    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027730    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027735    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.032850 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027740    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027744    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027748    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027753    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027757    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027761    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027765    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027769    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027774    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027778    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027783    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027787    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027791    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027796    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027800    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027804    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027811    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027815    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027820    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.033643 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027823    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027828    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027832    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027836    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027840    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027845    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027849    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027853    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027857    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027861    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027865    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027873    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027879    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.027884    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028806    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028817    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028822    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028827    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028831    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.034302 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028835    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028839    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028844    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028849    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028853    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028857    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028861    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028866    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028870    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028874    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028882    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028887    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028892    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028896    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028901    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028906    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028911    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028917    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028921    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.035043 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028925    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028930    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028935    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028939    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028944    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028948    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028952    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028956    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028961    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028965    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028970    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028975    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028979    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028983    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028987    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028991    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.028996    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029000    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029004    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.035782 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029008    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029012    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029017    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029021    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029027    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029031    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029037    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029041    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029046    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029050    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029054    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029058    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029062    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029067    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029071    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029075    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029080    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029084    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029088    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029092    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.036603 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029096    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029101    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029107    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029111    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029118    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029122    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029126    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029131    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029135    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029139    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029143    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029148    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029155    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029162    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029167    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029171    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029177    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029181    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029185    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.037352 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029190    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029194    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029198    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029203    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029208    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029212    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029217    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029221    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029225    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029229    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029234    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029238    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029243    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029247    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029252    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029256    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029261    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029267    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029270    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029275    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.038131 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029279    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029284    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029288    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.029292    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029431    5831 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029442    5831 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029451    5831 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029458    5831 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029466    5831 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029472    5831 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029479    5831 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029485    5831 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029490    5831 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029495    5831 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029500    5831 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029506    5831 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029511    5831 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029516    5831 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029520    5831 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029547    5831 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029552    5831 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029557    5831 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029565    5831 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029569    5831 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:07:30.038937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029575    5831 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029580    5831 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029586    5831 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029592    5831 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029597    5831 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029603    5831 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029609    5831 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029613    5831 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029618    5831 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029622    5831 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029628    5831 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029632    5831 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029640    5831 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029644    5831 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029649    5831 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029653    5831 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029658    5831 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029665    5831 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029673    5831 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029679    5831 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029684    5831 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029689    5831 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029694    5831 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029701    5831 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:07:30.039791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029706    5831 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029711    5831 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029725    5831 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029731    5831 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029735    5831 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029740    5831 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029744    5831 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029749    5831 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029753    5831 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029758    5831 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029769    5831 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029775    5831 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029780    5831 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029785    5831 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029792    5831 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029798    5831 flags.go:64] FLAG: --help="false"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029803    5831 flags.go:64] FLAG: --hostname-override="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029809    5831 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029814    5831 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029818    5831 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029824    5831 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029830    5831 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029834    5831 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:07:30.040701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029839    5831 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029844    5831 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029850    5831 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029855    5831 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029861    5831 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029865    5831 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029870    5831 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029874    5831 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029879    5831 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029884    5831 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029888    5831 flags.go:64] FLAG: --lock-file=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029893    5831 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029897    5831 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029902    5831 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029912    5831 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029917    5831 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029921    5831 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029926    5831 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029931    5831 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029936    5831 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029941    5831 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029945    5831 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029952    5831 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029958    5831 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029965    5831 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:07:30.041646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029970    5831 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029976    5831 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029981    5831 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029985    5831 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029990    5831 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029995    5831 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.029999    5831 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030010    5831 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030014    5831 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030023    5831 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030029    5831 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030033    5831 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030037    5831 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030042    5831 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030047    5831 flags.go:64] FLAG: --port="10250"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030052    5831 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030057    5831 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0681b6506b7fbaf66"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030063    5831 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030068    5831 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030073    5831 flags.go:64] FLAG: --register-node="true"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030078    5831 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030085    5831 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030089    5831 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030093    5831 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030098    5831 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030105    5831 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:07:30.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030110    5831 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030114    5831 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030119    5831 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030124    5831 flags.go:64] FLAG: --runonce="false"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030129    5831 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030134    5831 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030139    5831 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030145    5831 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030150    5831 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030155    5831 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030160    5831 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030165    5831 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030170    5831 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030174    5831 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030179    5831 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030186    5831 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030191    5831 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030195    5831 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030200    5831 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030209    5831 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030215    5831 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030220    5831 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030227    5831 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030232    5831 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030237    5831 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:07:30.043676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030242    5831 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030246    5831 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030251    5831 flags.go:64] FLAG: --v="2"
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030259    5831 flags.go:64] FLAG: --version="false"
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030265    5831 flags.go:64] FLAG: --vmodule=""
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030272    5831 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.030277    5831 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030497    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030506    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030511    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030515    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030519    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030542    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030547    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030553    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030560    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030567    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030573    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030579    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030584    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.044641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030589    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030594    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030599    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030603    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030608    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030612    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030617    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030624    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030628    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030633    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030637    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030641    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030645    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030649    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030654    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030658    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030662    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030666    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030670    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030674    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.045379 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030678    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030683    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030687    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030691    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030695    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030699    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030703    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030708    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030713    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030717    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030721    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030726    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030730    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030734    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030738    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030742    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030746    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030750    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030754    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030760    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030765    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.046347 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030769    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030774    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030777    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030782    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030786    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030790    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030794    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030799    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030804    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030808    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030812    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030816    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030821    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030825    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030829    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030833    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030838    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030842    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030847    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.047205 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030852    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030856    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030860    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030864    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030868    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030873    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030877    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030881    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030886    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030890    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030894    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030903    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030907    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030911    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030916    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030920    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030926    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030932    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030937    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.048136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030941    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030945    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030950    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030954    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030958    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030962    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030966    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030971    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030975    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030979    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030983    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030988    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030992    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.030996    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.031044    5831 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031103    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.049013 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031111    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031116    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031120    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031125    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031129    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031133    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031137    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031141    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031148    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031152    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031157    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031161    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031165    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031170    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031174    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031178    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031182    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031187    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031191    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031195    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.049706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031202    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031206    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031211    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031215    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031219    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031224    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031228    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031232    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031236    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031240    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031245    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031251    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031255    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031259    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031264    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031268    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031272    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031278    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031283    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.050649 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031288    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031295    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031299    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031303    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031307    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031311    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031316    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031320    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031324    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031329    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031333    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031338    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031342    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031347    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031352    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031356    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031360    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031364    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031368    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.051480 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031372    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031376    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031380    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031385    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031391    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031397    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031404    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031409    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031413    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031418    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031423    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031427    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031432    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031436    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031443    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031449    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031453    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031457    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031462    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.052297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031466    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031470    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031474    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031478    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031482    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031487    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031491    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031497    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031502    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031506    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031510    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031514    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031518    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031522    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031548    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031552    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031556    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031560    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031564    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031567    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.053163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031572    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031576    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031580    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031584    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031589    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031593    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031597    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.031603    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.047043    5831 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.047069    5831 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.047102    5831 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.047109    5831 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047202    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047210    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047215    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047219    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047224    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047228    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047232    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.054119 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047237    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047241    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047246    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047251    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047256    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047260    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047265    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047270    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047274    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047278    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047282    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047286    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047291    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047295    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047300    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047304    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047309    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047313    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047317    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047321    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.054706 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047327    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047332    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047337    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047341    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047345    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047350    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047354    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047358    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047362    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047367    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047371    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047375    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047380    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047384    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047388    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047393    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047397    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047401    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047405    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047419    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.055246 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047424    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047428    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047432    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047437    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047442    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047446    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047450    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047455    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047459    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047463    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047467    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047475    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047481    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047486    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047491    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047495    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047499    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047503    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047508    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.055940 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047513    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047517    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047522    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047544    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047548    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047552    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047556    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047560    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047563    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047567    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047571    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047575    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047579    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047591    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047595    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047598    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047602    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047606    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047610    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.056552 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047614    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047619    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047624    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047628    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047632    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047636    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047641    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047646    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047650    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047654    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047658    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047662    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047667    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047671    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047678    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047684    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047690    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047695    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047700    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.057031 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047704    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.047709    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.047761    5831 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048009    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048017    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048024    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048028    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048033    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048039    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048043    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048047    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048052    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048056    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048061    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048065    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048069    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.057659 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048073    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048078    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048082    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048087    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048092    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048096    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048100    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048104    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048108    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048113    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048117    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048121    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048125    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048130    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048133    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048137    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048141    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048145    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048150    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048154    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.058155 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048158    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048162    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048167    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048171    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048178    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048186    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048192    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048197    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048202    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048211    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048216    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048221    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048225    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048229    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048233    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048238    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048243    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048247    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048251    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.058972 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048255    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048259    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048264    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048268    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048272    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048276    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048280    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048285    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048289    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048293    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048297    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048301    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048305    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048310    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048315    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048319    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048323    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048328    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048332    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048337    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.059464 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048341    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048346    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048352    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048357    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048361    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048365    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048369    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048373    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048378    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048383    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048387    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048391    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048395    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048399    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048404    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048408    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048412    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048417    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048421    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048425    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.060036 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048430    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048435    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048441    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048447    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048452    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048459    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048464    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048468    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048472    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048476    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048481    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048485    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048490    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048495    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.048562    5831 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:30.060548 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048619    5831 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048625    5831 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048630    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048635    5831 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048640    5831 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048644    5831 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048649    5831 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048653    5831 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048657    5831 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048661    5831 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048666    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048671    5831 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048675    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048679    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048683    5831 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048688    5831 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048692    5831 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048696    5831 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048700    5831 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:30.060966 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048704    5831 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048709    5831 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048713    5831 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048720    5831 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048725    5831 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048729    5831 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048734    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048738    5831 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048742    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048747    5831 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048751    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048757    5831 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048761    5831 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048767    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048772    5831 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048776    5831 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048780    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048784    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048788    5831 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048793    5831 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:30.061431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048797    5831 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048802    5831 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048806    5831 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048810    5831 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048815    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048819    5831 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048823    5831 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048827    5831 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048831    5831 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048836    5831 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048840    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048844    5831 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048849    5831 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048853    5831 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048859    5831 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048867    5831 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048872    5831 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048877    5831 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048881    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048886    5831 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:30.062045 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048890    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048895    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048899    5831 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048903    5831 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048909    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048916    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048920    5831 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048924    5831 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048928    5831 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048933    5831 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048937    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048942    5831 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048946    5831 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048951    5831 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048955    5831 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048959    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048963    5831 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048967    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048971    5831 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:30.062756 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048976    5831 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048981    5831 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048986    5831 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048991    5831 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048995    5831 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.048999    5831 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049003    5831 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049008    5831 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049014    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049018    5831 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049022    5831 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049026    5831 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049031    5831 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049035    5831 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049039    5831 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049044    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049048    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049053    5831 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049059    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049064    5831 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:30.063424 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049068    5831 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049072    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049077    5831 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049081    5831 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049089    5831 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049097    5831 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049103    5831 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:30.049107    5831 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.049202    5831 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.050826    5831 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.050907    5831 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.051202    5831 server.go:1014] "Starting client certificate rotation"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.051368    5831 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.051426    5831 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:07 +0000 UTC" deadline="2028-07-21 00:37:24.805057806 +0000 UTC"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.051476    5831 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15093h29m54.753585349s"
Oct 31 03:07:30.064269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.051982    5831 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:30.064785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.053881    5831 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:30.064785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.055631    5831 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:07:30.064785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.057675    5831 log.go:25] "Validated CRI v1 image API"
Oct 31 03:07:30.064785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.058476    5831 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:07:30.064785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.060946    5831 fs.go:135] Filesystem UUIDs: map[40e51575-7ee4-4bdf-b66f-a3b87d546dc7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a4de9143-c1d0-4490-a9b2-79605ef3b55e:/dev/nvme0n1p3]
Oct 31 03:07:30.064974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.060963    5831 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2654e96dcd4b70e73a1fd1beca3ab3a9c833bc686b30a9079993f16cfe369fe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2654e96dcd4b70e73a1fd1beca3ab3a9c833bc686b30a9079993f16cfe369fe3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cd4c0559b736a091506203218322055a7d05526cd14e51c68266db5db133594/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cd4c0559b736a091506203218322055a7d05526cd14e51c68266db5db133594/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31a110bcda188a202bbc102cf068e93f628b4a13a46db8fb1864e23aff1cb41e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31a110bcda188a202bbc102cf068e93f628b4a13a46db8fb1864e23aff1cb41e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/557f33a737c3bc45193294dbc9136f5e6058eb9a041b5b2c5c84f43318ed5edc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/557f33a737c3bc45193294dbc9136f5e6058eb9a041b5b2c5c84f43318ed5edc/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e91450e567c87cc9e49a9454ab7ce248a7bfd0079c7e3b92f119d5a85cdc67b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e91450e567c87cc9e49a9454ab7ce248a7bfd0079c7e3b92f119d5a85cdc67b/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d/userdata/shm major:0 minor:454 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~projected/kube-api-access-7p4jz:{mountpoint:/var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~projected/kube-api-access-7p4jz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01f13667-aa82-4f8e-b860-abcfb602fc4c/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/01f13667-aa82-4f8e-b860-abcfb602fc4c/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~projected/kube-api-access-4lzj8:{mountpoint:/var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~projected/kube-api-access-4lzj8 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~projected/kube-api-access-qxt8k:{mountpoint:/var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~projected/kube-api-access-qxt8k major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~projected/kube-api-access-fdpgl:{mountpoint:/var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~projected/kube-api-access-fdpgl major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27fe767b-0eac-42e2-a937-e13497bad6bd/volumes/kubernetes.io~projected/kube-api-access-zxk4t:{mountpoint:/var/lib/kubelet/pods/27fe767b-0eac-42e2-a937-e13497bad6bd/volumes/kubernetes.io~projected/kube-api-access-zxk4t major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a46e66-bcbb-4bc9-b57b-deb78bc76adf/volumes/kubernetes.io~projected/kube-api-access-5drl7:{mountpoint:/var/lib/kubelet/pods/37a46e66-bcbb-4bc9-b57b-deb78bc76adf/volumes/kubernetes.io~projected/kube-api-access-5drl7 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f85eaf9-d372-4e12-8016-0ade17b5badc/volumes/kubernetes.io~projected/kube-api-access-c97wg:{mountpoint:/var/lib/kubelet/pods/4f85eaf9-d372-4e12-8016-0ade17b5badc/volumes/kubernetes.io~projected/kube-api-access-c97wg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/kube-api-access-fkp8k:{mountpoint:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/kube-api-access-fkp8k major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/registry-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fb14716-1ea1-40f5-9ac7-7476c9daa35b/volumes/kubernetes.io~projected/kube-api-access-zrqzr:{mountpoint:/var/lib/kubelet/pods/5fb14716-1ea1-40f5-9ac7-7476c9daa35b/volumes/kubernetes.io~projected/kube-api-access-zrqzr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5feec31a-0c22-428a-9d9a-3c16b3cd6097/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5feec31a-0c22-428a-9d9a-3c16b3cd6097/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71bad7a7-3b59-4e50-96d9-59dcd2ec790c/volumes/kubernetes.io~projected/kube-api-access-5596k:{mountpoint:/var/lib/kubelet/pods/71bad7a7-3b59-4e50-96d9-59dcd2ec790c/volumes/kubernetes.io~projected/kube-api-access-5596k major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c7f2a8-6907-4ca4-b636-e0807b250ef9/volumes/kubernetes.io~projected/kube-api-access-j2wd6:{mountpoint:/var/lib/kubelet/pods/78c7f2a8-6907-4ca4-b636-e0807b250ef9/volumes/kubernetes.io~projected/kube-api-access-j2wd6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb3c50f-ca36-437c-b720-034fa17cb3e1/volumes/kubernetes.io~projected/kube-api-access-mpkhl:{mountpoint:/var/lib/kubelet/pods/7cb3c50f-ca36-437c-b720-034fa17cb3e1/volumes/kubernetes.io~projected/kube-api-access-mpkhl major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801996f5-e5f7-4ef2-93b0-ce1b36c94343/volumes/kubernetes.io~projected/kube-api-access-7qxfp:{mountpoint:/var/lib/kubelet/pods/801996f5-e5f7-4ef2-93b0-ce1b36c94343/volumes/kubernetes.io~projected/kube-api-access-7qxfp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~projected/kube-api-access-jxz8t:{mountpoint:/var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~projected/kube-api-access-jxz8t major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/984c8e10-fc2b-4012-9b5b-78b9e3770178/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/984c8e10-fc2b-4012-9b5b-78b9e3770178/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bfd162-e42e-423a-907c-1ae68fa18ec7/volumes/kubernetes.io~projected/kube-api-access-49g4k:{mountpoint:/var/lib/kubelet/pods/b5bfd162-e42e-423a-907c-1ae68fa18ec7/volumes/kubernetes.io~projected/kube-api-access-49g4k major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~projected/kube-api-access-gw64p:{mountpoint:/var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~projected/kube-api-access-gw64p major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~secret/metrics-certs major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfe907fb-0ddb-48d8-8a02-8ee398870770/volumes/kubernetes.io~projected/kube-api-access-v2pd4:{mountpoint:/var/lib/kubelet/pods/bfe907fb-0ddb-48d8-8a02-8ee398870770/volumes/kubernetes.io~projected/kube-api-access-v2pd4 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~projected/kube-api-access-92xxs:{mountpoint:/var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~projected/kube-api-access-92xxs major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~secret/metrics-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9/volumes/kubernetes.io~projected/kube-api-access-b46bj:{mountpoint:/var/lib/kubelet/pods/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9/volumes/kubernetes.io~projected/kube-api-access-b46bj major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~projected/kube-api-access-5nmwv:{mountpoint:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~projected/kube-api-access-5nmwv major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/default-certificate major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/stats-auth major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b1e5a2-6307-4472-9502-5e97c55ac76f/volumes/kubernetes.io~projected/kube-api-access-5xtvl:{mountpoint:/var/lib/kubelet/pods/d9b1e5a2-6307-4472-9502-5e97c55ac76f/volumes/kubernetes.io~projected/kube-api-access-5xtvl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~projected/kube-api-access-trtrq:{mountpoint:/var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~projected/kube-api-access-trtrq major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc279cb7-148d-4787-983c-ea212b73a3cc/volumes/kubernetes.io~projected/kube-api-access-m7nqz:{mountpoint:/var/lib/kubelet/pods/dc279cb7-148d-4787-983c-ea212b73a3cc/volumes/kubernetes.io~projected/kube-api-access-m7nqz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8ddf63-307d-4908-bff6-df337941dfaf/volumes/kubernetes.io~projected/kube-api-access-gtm42:{mountpoint:/var/lib/kubelet/pods/dc8ddf63-307d-4908-bff6-df337941dfaf/volumes/kubernetes.io~projected/kube-api-access-gtm42 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ead07c5f-c2fb-42de-9254-059fab1e0ab7/volumes/kubernetes.io~projected/kube-api-access-gcrhv:{mountpoint:/var/lib/kubelet/pods/ead07c5f-c2fb-42de-9254-059fab1e0ab7/volumes/kubernetes.io~projected/kube-api-access-gcrhv major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~projected/kube-api-access-zps7s:{mountpoint:/var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~projected/kube-api-access-zps7s major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~projected/kube-api-access-vkv7s:{mountpoint:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~projected/kube-api-access-vkv7s major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~projected/kube-api-access-dl4cb:{mountpoint:/var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~projected/kube-api-access-dl4cb major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~projected/kube-api-access-gpmhz:{mountpoint:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~projected/kube-api-access-gpmhz major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3baecd3fbf4619600bedbe442df9bbc7200589a0fc0339decff77e90a142b18c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5b1e9f4a27a82c9d8d80f9d2c11b00672272ce13d8886ea30befd54ac9e580c4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d8fcf7b45ada1bf434c27807c22000b4c22f8347a81a40db09665ec2073bddc3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/87ab8a5df2266c5f514bbe1ac166f75e49b56ed536bdd8f3b30cacddab9abfe9/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ff4130233ecf0c0ed63bf1d8b9ff3d7009d56aa98270a0d6afb874f7058253bd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c58d0b74b9c1bd7afd51c9d4367a0c8620d4a159db2b48113ef6fc722e6f9278/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f8582ec8ccbfff9c1d3c6cfe89b23b47b28d1fba4ece4ba2747713d50d5cea63/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b5e5e40cbbc51ef07749a5ebfb9e6303d6e7cb17c465a3a8649ecede05518d2f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ee1005ed82de1ff08f2f8df335388f6ffe2e423c2efe2f9d477492216a1c164a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c4f2e2bdd91e158405afd0983ebbb25ef3e73bdd80d415a0a4f9a90f9e632e64/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/fa80a6fed3352660ec8668bd7065a68e07a281dbd35ec07fb6591afb2f73427d/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/52423acad144c0bc3a01a25062c336f0a84123b2b44071e257793c8aa4507daa/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/85a85e776071e558350f4d7692452bc4ca6bffb4e95ad24dc10a4fbb0d11a395/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0eff477e18c26156a9002ec4e7024e01f4f0e842e25f61169de414274f6615c1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/570c48149722fb27dfa32dd8ac99393fe07cecea0ccae633e58ccddda6172a16/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6c533b88949aa6906d03679b39336e882630ddaf7b44913ef9d02d339ce1f588/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ae625e07ba5134d19c7a971715d6749edae7321ad1e1b294d9f4080ce3c8d7b0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6e78bc7761bf80f812276a3779cf0a4b3a78befa897fca49560c5cb2379824ba/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c321e891ad28fe282b981acb1e4a7492681371bdc71fd016da4e430e9b492bd6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/638de991af66d59f3a31b27fd92f0a1d4e9b5c07494a4dc2e3f0b6b2586d8944/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6be5c9fc1083dafda17ddeaffdec19d2ef8ed2935b09c0cf3486538bccaa409c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/f59026e2336d8086531c466c8d08c533569797bdb0c6caca1a3f4f06e59adf42/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3a03088326873ee12e722acdc1f24d289a3e483596ea2951a1ccefe8bd5d5a63/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/2d3ca48bf23b722379cbc76c13a4184ed0460059ec23e53515c63a66f7a9d909/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/965606be48f883d1699d533d3178c5176e17fb6fedf2a597161d0725354130bf/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b4f10f86451c1d73f03b44e7cb248554b8ec8f0b75189e027dfbbe6134e54a34/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d3c58ef81fa572201a72f3080e92023f0d9de573a4036dc77ebc57d818ecc313/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/e0d8b098fd5a3250aea1d4b04e045d02c54ed5445fcbc7515ab1d47e9f9835c6/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/6a5c222c1c38b710e9deb827e4ee82f77a108a2cbd56f34ca792e9d664d6f96c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/049c7838eebb0629859ff563701ec9d42ef862270c737047a000412dcb88ae91/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/a159ca05e690b49717a1debc0b0e927d3d76fd4b2e8c0264dc2b46845c003286/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d73de61ddb4eab08259c8c363ecc5c46ce92707157a8d5e793c846136e62aefb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/6523a2b5ce44004debc5da0691ec6776db21c258fc79c0f755e4eecd6125f5c1/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/eb4dabbb920ce61078d40a315051c69374766b5d013146f850d1847416461223/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/14987966a266244d2dd0e52d5a34552d89978adee4f87993da49167f8cd7df8b/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/6f612e00cdab0426e50a6e14697a555a426aa582c187f84d03bbcf019e35369a/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/2739afcb72fb41cb0a623308b07d0f2877bd4daf1ea45960486803bb2ff18e5d/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/7357cf0e9008f76fee7b14b721cdee7bec3c2b200b2f9a6c2a57b5c8e98956a8/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/d803afbb64562d4b97ba985a3a5c60f8b42a7bf9c0a691cccd872bfaaa941b12/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/bbb7ba58b91f7cdd41d61fc28d3c3d2b85af57f091904df80c21a1c1445f6004/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/0ee8c4423c342e95eb2aeadf3441893db174f7de94b388a046d845d2383a2058/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/ea421b85ec73e2656a23012ccb389b1f5600411770b093435dce63a19c0a206f/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/65e60eb1d8c25a71ed1a202a3e8562a77cc6fd6a56b15fbfb9fb465dd3b0862e/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/734f46b4d0c817c1b9e0d7fe5146ae908786d83152bf1c9faeb843e87aad04b9/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/8ca837c2470af3f01270d895d1edde5f37392b667dfa010cccf3f0c606ce4424/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/24473657502f0f7bc2d34f804b7d737f381ba4c67b4ec785adeea3235a4c1436/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/4b34729475332b31e767734a759d8fcb5dac41543770a0155edc447434874e53/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/79b461bf223bc19988aa0e71003c027f00f8d9d309d9a7cabe67bf64dfb29bb7/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/724df5ad0deb1e4f083082b0a4c772c6dca343677c8a1c1272c4bc1f35bb6e8f/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/e0639e10b98ba43f5b59bf73d5667573ce29c7631bcd9eb5d4f0882a6b8f4c3e/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/b1ab9edc73777558e208925c06f8e01dfd2ee0a7ccc02daf24f5172296c58eff/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/ef3e6d5efa8f260cbfc4fb5c666b2d6f451728fe26c7bd7c05d8d554297465c1/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/fc4da1e291fe6d7d29ae9cd5c2f49076ada8d4b9ff6b9e24ed87c579d7be0224/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/debfe8e1abb43508a1edbb732184d1d1d0f95ffc651b1531e097281c3451f6cf/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/0325b674620fe81d7658390fc0ed70cad7b2851c734f36930a0e4541ffc2f90f/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/cbbece7ae16407ebac605699bfda7f1e42bd4552b87fa9b51301192053b0cdda/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/2e3913f0ba8ac0bf4c699abc2833e7f321483c4cf444eca2c396bb795098b9ce/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/4181ea02ae8775f36e0f260ecc6fcf53f590ef8ee2e68987ebdf28472a675bfb/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/f78b0d762109154b40ead6ba1fa8a34bd6e5ab7391753e8e11cf4417fc94429b/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/ab4baca47cfeb73024b36c4939a891b33410957f9245ace514db9a8985910d9a/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/1c46aa616bbc18361524de50e94f9da92f45fceaa39224081899213a52ff40d9/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/69eb0c75968e6b3c07ed2d2591dbe6e2434bf175c1478f5bdd4496c82c5ac128/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/c342114e4336b3bec9eadcb7fbf34f2f129b469394894c8d66b917571b9e0e49/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/11c9503611fa5c372f2bb876453aa2c3274d39b847aa145b16b1ca18ac39e0b6/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/d2620bc86e4692290ea2a8b8da20286f033a388f4f21704ce19bd18c831a9bbd/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/ef508c4d8d9d518d7f3f830ec729c1a9aa6d20a11e4ea04e99c018b6d225eb0a/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/d72a6c468b1e082aad00f259e8a86506c5131fbc6fc4ee85c886075fde07613a/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/e38262ededa25ada2fe1a9ab8d19fcf8a3fa6fbe4e6c84c6bf77a9a8172081e2/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/3412ff29726a258c1b3f0e5b0021a37bd003bcac2edc78305447509a9a805e72/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/b451b6163c3a4debbe68a69653d2585a8aa04d6a0e04f7d030b46a4047da5cd0/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/effe525f9b03f6406a4594b09ef49f5955c18ef0f74db8b63f32798b0e75332f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5d5d319d842c5950aeafcbb281c4a555c723e34cc87d08c7b4783ef383405aa8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/09523041e0becdbb710956b2f2d44397450a4e3ce5c646e5fad657d4daf5a090/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/5c3d6790ae5f01c6eb5564732c1036396c01457ac0a7c6e2d35379b34504b7a5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e7a91fe2aed810a6503d9fd27f92c6f4a430b1f31dbd453a85771219b0859e55/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/114cc303088e2ba392e2c63c151a40adb24ae70d3219022bf30d691e78c619d4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/441d29ce2fcc5bb9425624a115a2196e8b893bbb9e679e53e6e9c38f88583277/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1a380238a4f9c3a8f23a4c1c08d44afcd7aa596980c848cc2565e0075c83f528/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a826c89a5e14e03de64cc8b5c61739064601eb2d23dae8ed302d6b20f973c2e1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/8269316297fb78c1c56f1a27b650d70ddceb8a315c24b6f04db6abebad76acdf/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 03:07:30.071119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.070699    5831 manager.go:217] Machine: {Timestamp:2026-10-31 03:07:30.070203582 +0000 UTC m=+0.084765287 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099353 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f04d97c0d665ac2f20c0c7a07bad SystemUUID:ec28f04d-97c0-d665-ac2f-20c0c7a07bad BootID:80283683-6a38-4ffb-9591-371bde0aceb7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b99c7d4b71cbedfbd7aec022fbe8e7729f7b101ce6e3a90e81eed82ff505df2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ead07c5f-c2fb-42de-9254-059fab1e0ab7/volumes/kubernetes.io~projected/kube-api-access-gcrhv DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~projected/kube-api-access-fdpgl DeviceMajor:0 DeviceMinor:327 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b1e5a2-6307-4472-9502-5e97c55ac76f/volumes/kubernetes.io~projected/kube-api-access-5xtvl DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~projected/kube-api-access-jxz8t DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afef3c4d52e8be96f5d54ee25d550ea6270e46e14da79c4606161d3957ac0b4b/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d6a731d42933bb40613a029613cd62d43b714acfc1a66333f77dee13dfb905d/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56c53ac4a083b979146c36d2057244d048fe7d2176dc8fc4a688d53158267c46/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ec5983afa46cef1a26158a2426d3e589b28172594da160d047c7d052722120e/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f43764ae1acb54841c52bfa867b80492372fe0f8c596b9733e28865d452285b2/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88d0ddb1-2f0c-451f-9909-846980b61b9f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8ddf63-307d-4908-bff6-df337941dfaf/volumes/kubernetes.io~projected/kube-api-access-gtm42 DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/37a46e66-bcbb-4bc9-b57b-deb78bc76adf/volumes/kubernetes.io~projected/kube-api-access-5drl7 DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/984c8e10-fc2b-4012-9b5b-78b9e3770178/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31a110bcda188a202bbc102cf068e93f628b4a13a46db8fb1864e23aff1cb41e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1efb3fa1fb69a06f7af77f16c1ab703b99669b306fc15e7a87853905930faff/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf66e82ea3d15fd00f3313c83ee80461e6eda468cf0dab697ec32e3062fc5e24/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c7f2a8-6907-4ca4-b636-e0807b250ef9/volumes/kubernetes.io~projected/kube-api-access-j2wd6 DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/009cba45-9f0a-4991-8ab1-86b7ed158f79/volumes/kubernetes.io~projected/kube-api-access-7p4jz DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bfd162-e42e-423a-907c-1ae68fa18ec7/volumes/kubernetes.io~projected/kube-api-access-49g4k DeviceMajor:0 DeviceMinor:316 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a13dc5b44ab42e9304a04cba38ab19e4d471cf3747f085c66196702f1d5d8d1d/userdata/shm DeviceMajor:0 DeviceMinor:454 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~projected/kube-api-access-zps7s DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5feec31a-0c22-428a-9d9a-3c16b3cd6097/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:402 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~projected/kube-api-access-gpmhz DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01f13667-aa82-4f8e-b860-abcfb602fc4c/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27fe767b-0eac-42e2-a937-e13497bad6bd/volumes/kubernetes.io~projected/kube-api-access-zxk4t DeviceMajor:0 DeviceMinor:231 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dc279cb7-148d-4787-983c-ea212b73a3cc/volumes/kubernetes.io~projected/kube-api-access-m7nqz DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~projected/kube-api-access-vkv7s DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfe907fb-0ddb-48d8-8a02-8ee398870770/volumes/kubernetes.io~projected/kube-api-access-v2pd4 DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cd4c0559b736a091506203218322055a7d05526cd14e51c68266db5db133594/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcf3e88085f71c174dcb881b144999593023f34b307e17aa46a44fb3d113e956/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e35970cc2d102ce357626a9da257012643bb257e31ef73f60973f9742ba93146/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/540b3dbeafd5083db1eb29ef6b4f5be2911062a6830104d234516d41cfa12cf9/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b41605b81e7ef9711079bff89c926defd70f80678f22b310902b1cfe8db01ea/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff779f60-ee44-48e3-b1fa-777374b9788b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e91450e567c87cc9e49a9454ab7ce248a7bfd0079c7e3b92f119d5a85cdc67b/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9450837d15366e125e0990b67ec2c609eb4ea5739c717409058794537340851a/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ec33cd8c-1d9c-437b-8742-661d57128951/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:363 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38ebac6fef92c8f5765b2cdffc8de31719aa8d9e566998c5a03a7e6fc20a1b4a/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a992d63c628da4ccefcea441c024a0942a1d00e05225b8aa2eb5cea45c1d264a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/625628064f59acd151201b6dbbd107fe44b0bd7ac537d97531f4f9cda19e8764/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/219ae079-d5b4-4aed-bf97-4766b3a2325d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:450 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b558d27-fb04-4431-88a3-22d27045dacb/volumes/kubernetes.io~projected/kube-api-access-qxt8k DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c860f4f881442911d5cace084c4af99a887d44fe67f6e4702b44d0995d91d136/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972/volumes/kubernetes.io~projected/kube-api-access-92xxs DeviceMajor:0 DeviceMinor:228 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:368 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2654e96dcd4b70e73a1fd1beca3ab3a9c833bc686b30a9079993f16cfe369fe3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9/volumes/kubernetes.io~projected/kube-api-access-b46bj DeviceMajor:0 DeviceMinor:311 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f85eaf9-d372-4e12-8016-0ade17b5badc/volumes/kubernetes.io~projected/kube-api-access-c97wg DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb3c50f-ca36-437c-b720-034fa17cb3e1/volumes/kubernetes.io~projected/kube-api-access-mpkhl DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf655c2e00ca3d36ff0a43d78a2a5060f19140bcb3ff241c9ebf301947b09126/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b8f2b00f-643e-412d-9ecb-94f0fd75cf29/volumes/kubernetes.io~projected/kube-api-access-gw64p DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5fb14716-1ea1-40f5-9ac7-7476c9daa35b/volumes/kubernetes.io~projected/kube-api-access-zrqzr DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/965950356c99ea980335f810e889b0fda741e7d3a61201f0a5991ce7898c9897/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d62e7b0da0cf4b31c000b1429d8c5c354a9d6d1a73d145b684a0018d5aaa3c82/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c9d56c992016bb97bed3fc5786a24899f4fac693a23c2bf93f1cc4be41a28c7/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51209ac5fb41fc375343233d6cb4cce3ccd482df475388c4d6eac610afbd445b/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes/kubernetes.io~projected/kube-api-access-fkp8k DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/557f33a737c3bc45193294dbc9136f5e6058eb9a041b5b2c5c84f43318ed5edc/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71bad7a7-3b59-4e50-96d9-59dcd2ec790c/volumes/kubernetes.io~projected/kube-api-access-5596k DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/da80e5b0-536a-46a8-aba6-c89002fa216d/volumes/kubernetes.io~projected/kube-api-access-trtrq DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0381e7c6-4dc6-4a42-a0cf-276261adcb26/volumes/kubernetes.io~projected/kube-api-access-4lzj8 DeviceMajor:0 DeviceMinor:232 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3837fe802eeeb99ae398f3e56c35166d5035cd0fd5e4254272be210145b5463/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fde8fbc1-b8e0-49a5-9725-4586869dce6c/volumes/kubernetes.io~projected/kube-api-access-dl4cb DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f8dcdbcd-d8e5-4400-880b-c89f3362af1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d608bf35-2830-4486-a05f-ac59ddb3c8e6/volumes/kubernetes.io~projected/kube-api-access-5nmwv DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/801996f5-e5f7-4ef2-93b0-ce1b36c94343/volumes/kubernetes.io~projected/kube-api-access-7qxfp DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c9d56c992016bb MacAddress:72:93:b9:03:c4:d8 Speed:10000 Mtu:8901} {Name:1d6a731d42933bb MacAddress:56:2e:dd:24:f7:bb Speed:10000 Mtu:8901} {Name:2cd4c0559b736a0 MacAddress:16:08:30:79:37:5e Speed:10000 Mtu:8901} {Name:31a110bcda188a2 MacAddress:b6:b1:af:c5:49:b1 Speed:10000 Mtu:8901} {Name:38ebac6fef92c8f MacAddress:5a:7c:42:c9:eb:2b Speed:10000 Mtu:8901} {Name:51209ac5fb41fc3 MacAddress:52:42:d5:d5:cb:48 Speed:10000 Mtu:8901} {Name:540b3dbeafd5083 MacAddress:7a:9f:f2:d6:43:b2 Speed:10000 Mtu:8901} {Name:557f33a737c3bc4 MacAddress:ea:e0:82:53:9e:20 Speed:10000 Mtu:8901} {Name:56c53ac4a083b97 MacAddress:d2:fd:16:e9:16:21 Speed:10000 Mtu:8901} {Name:5e91450e567c87c MacAddress:da:45:b2:22:4f:16 Speed:10000 Mtu:8901} {Name:625628064f59acd MacAddress:06:43:01:d8:da:41 Speed:10000 Mtu:8901} {Name:92da36f0fdbbda3 MacAddress:de:05:0e:fb:1d:07 Speed:10000 Mtu:8901} {Name:9450837d15366e1 MacAddress:3e:b4:b4:1b:ef:b2 Speed:10000 Mtu:8901} {Name:9ec5983afa46cef MacAddress:12:4d:55:13:f9:e3 Speed:10000 Mtu:8901} {Name:a13dc5b44ab42e9 MacAddress:16:7f:55:4a:05:04 Speed:10000 Mtu:8901} {Name:afef3c4d52e8be9 MacAddress:c6:44:8e:41:c9:29 Speed:10000 Mtu:8901} {Name:bf66e82ea3d15fd MacAddress:de:d4:2c:d4:ab:07 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fd:96:65:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:02:2a:7e:c6:7f:58 Speed:0 Mtu:8901} {Name:c860f4f88144291 MacAddress:22:61:05:2d:fc:54 Speed:10000 Mtu:8901} {Name:cc2d1dde4ea5d9c MacAddress:82:14:35:2f:2f:29 Speed:10000 Mtu:8901} {Name:cf655c2e00ca3d3 MacAddress:fa:09:03:b5:c4:35 Speed:10000 Mtu:8901} {Name:e35970cc2d102ce MacAddress:b6:0b:cf:71:34:f0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fd:96:65:4d Speed:0 Mtu:9001} {Name:f1efb3fa1fb69a0 MacAddress:76:33:80:07:c8:b0 Speed:10000 Mtu:8901} {Name:f43764ae1acb548 MacAddress:56:2e:d4:51:58:6e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:2a:fd:e4:d6:09 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:c0:d4:91:5f:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:07:30.071119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071122    5831 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:07:30.071476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071167    5831 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:07:30.071476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071408    5831 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:07:30.071587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071434    5831 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:07:30.071637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071590    5831 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:07:30.071637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071598    5831 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:07:30.071637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071608    5831 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:30.071637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071627    5831 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:30.071743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071699    5831 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:07:30.071866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071856    5831 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:07:30.071970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071961    5831 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:07:30.072001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.071996    5831 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:07:30.072029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072008    5831 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:07:30.072029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072020    5831 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:07:30.072029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072029    5831 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:07:30.072116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072037    5831 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:07:30.072781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072764    5831 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:30.072781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.072779    5831 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:30.073275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.073254    5831 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:07:30.085693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.085673    5831 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:30.085693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.085688    5831 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:30.085818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.085807    5831 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 03:07:30.086051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086039    5831 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:07:30.086092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086070    5831 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:07:30.086179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086168    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:07:30.086212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086187    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:07:30.086243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086219    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:07:30.086243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086226    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:07:30.086243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086231    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:07:30.086243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086237    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:07:30.086243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086244    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086253    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086263    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086270    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086278    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086287    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086312    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:07:30.086426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.086318    5831 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:07:30.089946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.089932    5831 server.go:1263] "Started kubelet"
Oct 31 03:07:30.090682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.089984    5831 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:07:30.090682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.090070    5831 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:07:30.090682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.090300    5831 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:07:30.090599 ip-10-0-140-194 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:07:30.093961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.092010    5831 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:07:30.093961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.093343    5831 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096584    5831 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096604    5831 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096678    5831 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096662    5831 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:09 +0000 UTC" deadline="2028-04-11 14:19:09.662318422 +0000 UTC"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096693    5831 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096694    5831 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12683h11m39.56562792s"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.096686    5831 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:07:30.098096 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.096750    5831 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:07:30.098516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098348    5831 factory.go:55] Registering systemd factory
Oct 31 03:07:30.098516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098366    5831 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:07:30.098745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098702    5831 factory.go:153] Registering CRI-O factory
Oct 31 03:07:30.098745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098726    5831 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:07:30.098859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098781    5831 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:07:30.098859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098803    5831 factory.go:103] Registering Raw factory
Oct 31 03:07:30.098859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.098820    5831 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:07:30.099167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.099153    5831 manager.go:319] Starting recovery of all containers
Oct 31 03:07:30.100067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100011    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100071    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c7f2a8-6907-4ca4-b636-e0807b250ef9" volumeName="kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100087    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da80e5b0-536a-46a8-aba6-c89002fa216d" volumeName="kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100103    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec33cd8c-1d9c-437b-8742-661d57128951" volumeName="kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100118    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c7f2a8-6907-4ca4-b636-e0807b250ef9" volumeName="kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100130    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb3c50f-ca36-437c-b720-034fa17cb3e1" volumeName="kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100142    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d608bf35-2830-4486-a05f-ac59ddb3c8e6" volumeName="kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100159    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d608bf35-2830-4486-a05f-ac59ddb3c8e6" volumeName="kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs" seLinuxMountContext=""
Oct 31 03:07:30.100179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100174    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec33cd8c-1d9c-437b-8742-661d57128951" volumeName="kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100186    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff779f60-ee44-48e3-b1fa-777374b9788b" volumeName="kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100198    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100211    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100225    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="984c8e10-fc2b-4012-9b5b-78b9e3770178" volumeName="kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100239    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cf6d1f-3302-4a26-85c1-5a6a8a06f972" volumeName="kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100256    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccfbbf60-9b29-45f3-be5a-0f1282beb2e9" volumeName="kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100268    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b1e5a2-6307-4472-9502-5e97c55ac76f" volumeName="kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100280    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ead07c5f-c2fb-42de-9254-059fab1e0ab7" volumeName="kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100292    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" volumeName="kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100304    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0381e7c6-4dc6-4a42-a0cf-276261adcb26" volumeName="kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100317    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100330    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bfd162-e42e-423a-907c-1ae68fa18ec7" volumeName="kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100344    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cf6d1f-3302-4a26-85c1-5a6a8a06f972" volumeName="kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100356    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d608bf35-2830-4486-a05f-ac59ddb3c8e6" volumeName="kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100368    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc279cb7-148d-4787-983c-ea212b73a3cc" volumeName="kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100380    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100411    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01f13667-aa82-4f8e-b860-abcfb602fc4c" volumeName="kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100425    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27fe767b-0eac-42e2-a937-e13497bad6bd" volumeName="kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100439    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100458    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71bad7a7-3b59-4e50-96d9-59dcd2ec790c" volumeName="kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100474    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d0ddb1-2f0c-451f-9909-846980b61b9f" volumeName="kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100486    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100500    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff779f60-ee44-48e3-b1fa-777374b9788b" volumeName="kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100512    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff779f60-ee44-48e3-b1fa-777374b9788b" volumeName="kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100552    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8ddf63-307d-4908-bff6-df337941dfaf" volumeName="kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100565    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="009cba45-9f0a-4991-8ab1-86b7ed158f79" volumeName="kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100577    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b558d27-fb04-4431-88a3-22d27045dacb" volumeName="kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100589    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100602    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d608bf35-2830-4486-a05f-ac59ddb3c8e6" volumeName="kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100614    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100626    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100638    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b558d27-fb04-4431-88a3-22d27045dacb" volumeName="kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert" seLinuxMountContext=""
Oct 31 03:07:30.100631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100649    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8f2b00f-643e-412d-9ecb-94f0fd75cf29" volumeName="kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100661    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="219ae079-d5b4-4aed-bf97-4766b3a2325d" volumeName="kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100674    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b558d27-fb04-4431-88a3-22d27045dacb" volumeName="kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100687    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="219ae079-d5b4-4aed-bf97-4766b3a2325d" volumeName="kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100698    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d0ddb1-2f0c-451f-9909-846980b61b9f" volumeName="kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100712    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfe907fb-0ddb-48d8-8a02-8ee398870770" volumeName="kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100724    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" volumeName="kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100734    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b558d27-fb04-4431-88a3-22d27045dacb" volumeName="kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100747    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="219ae079-d5b4-4aed-bf97-4766b3a2325d" volumeName="kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100758    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f85eaf9-d372-4e12-8016-0ade17b5badc" volumeName="kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100770    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb3c50f-ca36-437c-b720-034fa17cb3e1" volumeName="kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100781    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" volumeName="kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100793    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="009cba45-9f0a-4991-8ab1-86b7ed158f79" volumeName="kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100804    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c7f2a8-6907-4ca4-b636-e0807b250ef9" volumeName="kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100817    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc279cb7-148d-4787-983c-ea212b73a3cc" volumeName="kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100839    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" volumeName="kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100850    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a46e66-bcbb-4bc9-b57b-deb78bc76adf" volumeName="kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100861    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb3c50f-ca36-437c-b720-034fa17cb3e1" volumeName="kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100872    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da80e5b0-536a-46a8-aba6-c89002fa216d" volumeName="kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100886    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec33cd8c-1d9c-437b-8742-661d57128951" volumeName="kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100897    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="009cba45-9f0a-4991-8ab1-86b7ed158f79" volumeName="kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100912    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01f13667-aa82-4f8e-b860-abcfb602fc4c" volumeName="kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100924    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100936    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8f2b00f-643e-412d-9ecb-94f0fd75cf29" volumeName="kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100947    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cf6d1f-3302-4a26-85c1-5a6a8a06f972" volumeName="kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100961    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc279cb7-148d-4787-983c-ea212b73a3cc" volumeName="kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100974    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" volumeName="kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100986    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="219ae079-d5b4-4aed-bf97-4766b3a2325d" volumeName="kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.100997    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fb14716-1ea1-40f5-9ac7-7476c9daa35b" volumeName="kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101008    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb3c50f-ca36-437c-b720-034fa17cb3e1" volumeName="kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101019    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801996f5-e5f7-4ef2-93b0-ce1b36c94343" volumeName="kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101031    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d608bf35-2830-4486-a05f-ac59ddb3c8e6" volumeName="kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101043    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b1e5a2-6307-4472-9502-5e97c55ac76f" volumeName="kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101055    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da80e5b0-536a-46a8-aba6-c89002fa216d" volumeName="kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101068    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fb14716-1ea1-40f5-9ac7-7476c9daa35b" volumeName="kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101081    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0381e7c6-4dc6-4a42-a0cf-276261adcb26" volumeName="kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101093    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b4593e-3511-4f4c-8dc6-b2599c75efe4" volumeName="kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101105    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5feec31a-0c22-428a-9d9a-3c16b3cd6097" volumeName="kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101117    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cf6d1f-3302-4a26-85c1-5a6a8a06f972" volumeName="kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101129    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101141    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0381e7c6-4dc6-4a42-a0cf-276261adcb26" volumeName="kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101153    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801996f5-e5f7-4ef2-93b0-ce1b36c94343" volumeName="kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101165    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="984c8e10-fc2b-4012-9b5b-78b9e3770178" volumeName="kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101177    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc279cb7-148d-4787-983c-ea212b73a3cc" volumeName="kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101194    5831 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde8fbc1-b8e0-49a5-9725-4586869dce6c" volumeName="kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101206    5831 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:07:30.102376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.101214    5831 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:07:30.112211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.112186    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:07:30.116139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.116121    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:07:30.116769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.116747    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:07:30.120190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.120171    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:07:30.125224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.125196    5831 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:07:30.126808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.126792    5831 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:07:30.126893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.126826    5831 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:07:30.126893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.126847    5831 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:07:30.127009 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.126887    5831 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:07:30.161286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.161259    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:07:30.161441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.161294    5831 generic.go:356] "Generic (PLEG): container finished" podID="1b558d27-fb04-4431-88a3-22d27045dacb" containerID="58111874d738c2f2ae92ad29969ee00461c0d4c1cd2e1925380595fa4e454010" exitCode=255
Oct 31 03:07:30.167675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167648    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="fcff57fdf4eb85e850aa82d52b26c66ca97472503954692e0dcab5bb11b97abe" exitCode=0
Oct 31 03:07:30.167675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167677    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="d2a8053a9f87a3d980733b15ed56c8f03256e386463ec8f0561ebced3abb5d9e" exitCode=0
Oct 31 03:07:30.167822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167686    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="f626fc7f352dce9dd4312d4324476612b3ffb39e3a21db29d858f90ea240cdfc" exitCode=0
Oct 31 03:07:30.167822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167692    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="ce11711ac92e52c9f45b7fc065be55ff20ab4924610fbfac10f82e937ff55d94" exitCode=0
Oct 31 03:07:30.167822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167704    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="c7a8b8a4cd5bb8bbd40bc51afe3351070d9990beba0b9bc10296cae46a563607" exitCode=0
Oct 31 03:07:30.167822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.167713    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc279cb7-148d-4787-983c-ea212b73a3cc" containerID="ab5ac43ac6f41c846f7f1ce51bc27546fcc18a8adc0f617c3676ebb67ab9428b" exitCode=0
Oct 31 03:07:30.177383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.177357    5831 generic.go:356] "Generic (PLEG): container finished" podID="9d001e32831032e1eb4e57b6d85f040b" containerID="00e4baa2d2740ef7da6062bdcfb04b69cd973b9fc1def1871d6b4af2c18876ed" exitCode=0
Oct 31 03:07:30.186675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.186651    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:07:30.187026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.187001    5831 generic.go:356] "Generic (PLEG): container finished" podID="f8dcdbcd-d8e5-4400-880b-c89f3362af1b" containerID="b2f4ae029a234d930f213693458bdb19d9f5b09883bfa9938ab25d4bac4034a3" exitCode=1
Oct 31 03:07:30.196817 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.196795    5831 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:07:30.215299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.215280    5831 manager.go:324] Recovery completed
Oct 31 03:07:30.227006 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.226978    5831 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 03:07:30.237210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.237194    5831 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:30.242098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.242078    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:30.242177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.242112    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:30.242177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.242124    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:30.244746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.244728    5831 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:07:30.244746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.244743    5831 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:30.244885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.244745    5831 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:07:30.244885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.244835    5831 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:07:30.244999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.244984    5831 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 03:07:30.245038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245000    5831 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 03:07:30.245038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245016    5831 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 03:07:30.245038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245022    5831 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 03:07:30.245038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245028    5831 policy_none.go:50] "Start"
Oct 31 03:07:30.245038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245036    5831 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:07:30.245179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245046    5831 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:07:30.245179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245175    5831 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 03:07:30.245241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245182    5831 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 03:07:30.245241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.245187    5831 policy_none.go:44] "Start"
Oct 31 03:07:30.246795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.246778    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:30.246861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.246807    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:30.246861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.246817    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:30.250318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.250294    5831 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:07:30.253054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253036    5831 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:07:30.253147 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.253066    5831 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:07:30.253147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253076    5831 server.go:87] "Starting device plugin registration server"
Oct 31 03:07:30.253389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253372    5831 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:07:30.253504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253391    5831 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:07:30.253504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253489    5831 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:07:30.253690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253653    5831 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:07:30.253748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253700    5831 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:07:30.253748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253708    5831 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:07:30.253852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.253756    5831 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:07:30.253688614Z","UUID":"5b8250f6-790e-40c9-844b-cebee0ec23b1","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:07:30.255087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.255067    5831 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:07:30.255180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.255111    5831 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:07:30.255575 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.255553    5831 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:07:30.255650 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.255599    5831 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-194.us-west-2.compute.internal\" not found"
Oct 31 03:07:30.268150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.268129    5831 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:30.268266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.268220    5831 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 03:07:30.270408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.270391    5831 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:30.270483 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.270415    5831 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:30.316831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.316807    5831 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:30.353806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.353737    5831 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:30.356688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.356671    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:30.356796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.356704    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:30.356796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.356719    5831 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-194.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:30.356796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.356783    5831 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.427840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.427786    5831 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"]
Oct 31 03:07:30.428476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428461    5831 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.428557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428511    5831 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.428557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428503    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerStarted","Data":"efb645d19e2e73f66246bcdd1e93448a518ee267725dfa4cab6f57f82137b2a2"}
Oct 31 03:07:30.428641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428575    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerDied","Data":"00e4baa2d2740ef7da6062bdcfb04b69cd973b9fc1def1871d6b4af2c18876ed"}
Oct 31 03:07:30.428641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"9d001e32831032e1eb4e57b6d85f040b","Type":"ContainerStarted","Data":"00413459bd2858d3579fa3598fea0e8fbbe3a863ec48244bdc3ab380932cf9df"}
Oct 31 03:07:30.428641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428602    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"d102acf3d8823a0d171d30a06e49c39c","Type":"ContainerStarted","Data":"35df32fda1ad850d5f9f1315e426a04ffab1332bedaf5d5eefced371f6a29dc7"}
Oct 31 03:07:30.428641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.428614    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal" event={"ID":"d102acf3d8823a0d171d30a06e49c39c","Type":"ContainerStarted","Data":"d1163dd53a7a973a380da8d45a49ae4a74f05acdb6918f282cd02af6a9cb4eac"}
Oct 31 03:07:30.461631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.461609    5831 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.461770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.461672    5831 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.495578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.495551    5831 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:30.495728 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.495608    5831 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.495912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.495897    5831 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:30.495948 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:30.495936    5831 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.570595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.570561    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.570595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.570599    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.570785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.570616    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.670995    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.671033    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.671050    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.671097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.671104    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d001e32831032e1eb4e57b6d85f040b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"9d001e32831032e1eb4e57b6d85f040b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:30.671292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:30.671142    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d102acf3d8823a0d171d30a06e49c39c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" (UID: \"d102acf3d8823a0d171d30a06e49c39c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:31.072384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.072355    5831 apiserver.go:52] "Watching apiserver"
Oct 31 03:07:31.168790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.168759    5831 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:07:31.169478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.169453    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-w9hwd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286","kube-system/kas-connection-checker-685895c7c9-675lv","openshift-image-registry/node-ca-9dj5j","openshift-ingress/router-default-6489c4dbb7-6rbbc","openshift-network-diagnostics/network-check-target-2nrm8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd","openshift-insights/insights-runtime-extractor-svcxz","openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk","kube-system/kas-connection-checker-685895c7c9-jjkm4","openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn","openshift-multus/network-metrics-daemon-s5fjf","openshift-insights/periodic-gathering-8xxvw-kh2n2","openshift-insights/insights-operator-7c4986d449-ldnrv","openshift-dns/dns-default-tbchd","kube-system/global-pull-secret-syncer-bj5qd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf","kube-system/konnectivity-agent-qhzmz","kube-system/kas-connection-checker-685895c7c9-qqbzc","kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j","openshift-multus/multus-additional-cni-plugins-m9b2j","openshift-cluster-node-tuning-operator/tuned-c72n4","openshift-ingress-canary/ingress-canary-ksv4j","openshift-multus/multus-r8762","openshift-network-operator/iptables-alerter-brjlb","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-tkdn9","openshift-network-console/networking-console-plugin-86c478bb88-dbpk4","openshift-image-registry/image-registry-5b48949c57-xszl6","openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv","openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk","openshift-dns/node-resolver-mlv4n"]
Oct 31 03:07:31.198389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.198352    5831 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:07:31.207734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.207712    5831 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:31.207904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.207825    5831 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:31.219717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.219694    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.219866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.219713    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 03:07:31.219866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.219749    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.220147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220124    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.220147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220137    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220171    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220190    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220203    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220244    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220279    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.220298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220285    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.220576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220323    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 03:07:31.220576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220334    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 03:07:31.220576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220367    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.220576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220368    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.220576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.220422    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 03:07:31.221041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221023    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 03:07:31.221163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221146    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 03:07:31.221221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221203    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.221221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221205    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 03:07:31.221399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221384    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.221466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221439    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 03:07:31.221466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221442    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.221589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221472    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 03:07:31.221589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.221482    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.265359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.265333    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 03:07:31.265482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.265467    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.266146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266126    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 03:07:31.266199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266133    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 03:07:31.266268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266250    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 03:07:31.266339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266314    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:07:31.266445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266348    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:31.266576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266560    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:07:31.266576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266572    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266584    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266621    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266644    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266655    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266667    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266661    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266691    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:07:31.266718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266699    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266737    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-77xsg\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266733    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266810    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266887    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266906    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nhzcr\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266930    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:07:31.267048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.266983    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:07:31.273406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273387    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.273510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273415    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw64p\" (UniqueName: \"kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:31.273510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273434    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.273510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273448    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.273510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273467    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.273751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273517    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.273751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273595    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.273751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273626    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxk4t\" (UniqueName: \"kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t\") pod \"network-check-source-6f78d59f99-2zqjk\" (UID: \"27fe767b-0eac-42e2-a937-e13497bad6bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:07:31.273751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273682    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.273751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273724    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273758    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273788    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273807    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-metrics-certs\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273817    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273835    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b558d27-fb04-4431-88a3-22d27045dacb-serving-cert\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273838    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-metrics-tls\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273843    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273899    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273913    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5feec31a-0c22-428a-9d9a-3c16b3cd6097-original-pull-secret\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273927    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273960    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.274000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273988    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pd4\" (UniqueName: \"kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4\") pod \"kas-connection-checker-685895c7c9-675lv\" (UID: \"bfe907fb-0ddb-48d8-8a02-8ee398870770\") " pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274012    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.273959    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-tmp-dir\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274039    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274020    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b1e5a2-6307-4472-9502-5e97c55ac76f-tmp-dir\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274065    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274113    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274123    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ec33cd8c-1d9c-437b-8742-661d57128951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274127    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274159    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4jz\" (UniqueName: \"kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wd6\" (UniqueName: \"kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274213    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274233    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274266    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274287    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da80e5b0-536a-46a8-aba6-c89002fa216d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274295    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmwv\" (UniqueName: \"kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274315    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274344    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274371    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274409    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrhv\" (UniqueName: \"kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv\") pod \"kas-connection-checker-685895c7c9-qqbzc\" (UID: \"ead07c5f-c2fb-42de-9254-059fab1e0ab7\") " pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274445    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274584    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274588    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274633    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274682    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274724    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274734    5831 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274757    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274758    5831 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.274799    5831 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274794    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.274841    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqzr\" (UniqueName: \"kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.274801    5831 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal"
Oct 31 03:07:31.275149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275054    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275207    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275270    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275372    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.275376    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275415    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275459    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275483    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtrq\" (UniqueName: \"kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275520    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxt8k\" (UniqueName: \"kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275617    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-trusted-ca\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275648    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.275701    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:31.7756646 +0000 UTC m=+1.790226309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275725    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275775    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4cb\" (UniqueName: \"kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275806    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275834    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxz8t\" (UniqueName: \"kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275859    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275886    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.275895    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.275907    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275912    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275940    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.275966    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:31.775950021 +0000 UTC m=+1.790511729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.275986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276050    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276079    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276105    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5596k\" (UniqueName: \"kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k\") pod \"kas-connection-checker-685895c7c9-jjkm4\" (UID: \"71bad7a7-3b59-4e50-96d9-59dcd2ec790c\") " pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276130    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276157    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276183    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxfp\" (UniqueName: \"kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276231    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276260    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276305    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nqz\" (UniqueName: \"kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-default-certificate\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276417    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276456    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276489    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-tmp\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276496    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.276616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276554    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276671    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276752    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276804    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.276856    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277193    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277203    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277234    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fde8fbc1-b8e0-49a5-9725-4586869dce6c-serving-cert\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277244    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzj8\" (UniqueName: \"kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277273    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277303    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277332    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde8fbc1-b8e0-49a5-9725-4586869dce6c-service-ca-bundle\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277441    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277476    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46bj\" (UniqueName: \"kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj\") pod \"migrator-66df5d496b-4tsxv\" (UID: \"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277505    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fde8fbc1-b8e0-49a5-9725-4586869dce6c-snapshots\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277511    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277523    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0381e7c6-4dc6-4a42-a0cf-276261adcb26-serving-cert\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277589    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/219ae079-d5b4-4aed-bf97-4766b3a2325d-data-volume\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277688    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xxs\" (UniqueName: \"kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277731    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277757    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277797    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv7s\" (UniqueName: \"kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277935    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277963    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.277997    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278032    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278075    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97wg\" (UniqueName: \"kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278108    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278143    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278177    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278207    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278239    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278271    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278300    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278309    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-metrics-certs\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278330    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.278441    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:31.278474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278509    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278619    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278656    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278511    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-tmp\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278792    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278827    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.278860    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:31.778841394 +0000 UTC m=+1.793403103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278917    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtm42\" (UniqueName: \"kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42\") pod \"volume-data-source-validator-58c9f5f46d-xc87v\" (UID: \"dc8ddf63-307d-4908-bff6-df337941dfaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.278953    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279087    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279129    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279163    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279202    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279237    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g4k\" (UniqueName: \"kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k\") pod \"migrator-66df5d496b-l4gzk\" (UID: \"b5bfd162-e42e-423a-907c-1ae68fa18ec7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279275    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279312    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279349    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zps7s\" (UniqueName: \"kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279354    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279385    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.279803    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.279873    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:31.779850198 +0000 UTC m=+1.794411903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279921    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtvl\" (UniqueName: \"kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279935    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-tuned\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279960    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.279992    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmhz\" (UniqueName: \"kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280027    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280061    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280110    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.280276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280259    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280303    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280338    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280343    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d608bf35-2830-4486-a05f-ac59ddb3c8e6-stats-auth\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpgl\" (UniqueName: \"kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280967    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.280984    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b558d27-fb04-4431-88a3-22d27045dacb-config\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01f13667-aa82-4f8e-b860-abcfb602fc4c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281253    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/984c8e10-fc2b-4012-9b5b-78b9e3770178-agent-certs\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281316    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281360    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281466    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d0ddb1-2f0c-451f-9909-846980b61b9f-cert\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281479    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281576    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281613    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281707    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.281779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281759    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281791    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.282021    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.281794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/009cba45-9f0a-4991-8ab1-86b7ed158f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.282041    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608bf35-2830-4486-a05f-ac59ddb3c8e6-service-ca-bundle\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.282109    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.282400    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec33cd8c-1d9c-437b-8742-661d57128951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.282639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.282481    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.319715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.319692    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:07:31.319868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.319725    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:07:31.319868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.319834    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:07:31.319987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.319952    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:07:31.319987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.319955    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320027    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:07:31.320154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320136    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320175    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320176    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:07:31.320426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320411    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.320487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320445    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320470    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320471    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:31.320633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320618    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.320680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320642    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mq4kc\""
Oct 31 03:07:31.320742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320730    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.320791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320779    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:07:31.320824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320803    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 03:07:31.320852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320832    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.320852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320841    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.320910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320858    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.320942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320930    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 03:07:31.320973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320940    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 03:07:31.320973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.320953    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321297    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321313    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321309    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321331    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321338    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321363    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321370    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321372    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321378    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321417    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321428    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321437    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:07:31.321448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321428    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321457    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321460    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321479    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321510    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321550    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321585    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321610    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321657    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321710    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321770    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.321941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.321823    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:07:31.322337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.322276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-config-volume\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.322615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.322577    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/801996f5-e5f7-4ef2-93b0-ce1b36c94343-serviceca\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.323880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.323865    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-config\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.324006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.323990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.324061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.324033    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0381e7c6-4dc6-4a42-a0cf-276261adcb26-config\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.324061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.324051    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovn-node-metrics-cert\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.325014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.324999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.325056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325028    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-env-overrides\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.325056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325046    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/984c8e10-fc2b-4012-9b5b-78b9e3770178-konnectivity-ca\") pod \"konnectivity-agent-qhzmz\" (UID: \"984c8e10-fc2b-4012-9b5b-78b9e3770178\") " pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:07:31.325705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325681    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pd4\" (UniqueName: \"kubernetes.io/projected/bfe907fb-0ddb-48d8-8a02-8ee398870770-kube-api-access-v2pd4\") pod \"kas-connection-checker-685895c7c9-675lv\" (UID: \"bfe907fb-0ddb-48d8-8a02-8ee398870770\") " pod="kube-system/kas-connection-checker-685895c7c9-675lv"
Oct 31 03:07:31.325809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325772    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4jz\" (UniqueName: \"kubernetes.io/projected/009cba45-9f0a-4991-8ab1-86b7ed158f79-kube-api-access-7p4jz\") pod \"cluster-samples-operator-6ddb6c8b94-26sqf\" (UID: \"009cba45-9f0a-4991-8ab1-86b7ed158f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf"
Oct 31 03:07:31.325873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325855    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-ovnkube-script-lib\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.325964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.325947    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01f13667-aa82-4f8e-b860-abcfb602fc4c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dbpk4\" (UID: \"01f13667-aa82-4f8e-b860-abcfb602fc4c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dbpk4"
Oct 31 03:07:31.326115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326100    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-iptables-alerter-script\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.326213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326193    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da80e5b0-536a-46a8-aba6-c89002fa216d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.326461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326437    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmwv\" (UniqueName: \"kubernetes.io/projected/d608bf35-2830-4486-a05f-ac59ddb3c8e6-kube-api-access-5nmwv\") pod \"router-default-6489c4dbb7-6rbbc\" (UID: \"d608bf35-2830-4486-a05f-ac59ddb3c8e6\") " pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:31.326577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326561    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqzr\" (UniqueName: \"kubernetes.io/projected/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-kube-api-access-zrqzr\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.326648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326575    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrhv\" (UniqueName: \"kubernetes.io/projected/ead07c5f-c2fb-42de-9254-059fab1e0ab7-kube-api-access-gcrhv\") pod \"kas-connection-checker-685895c7c9-qqbzc\" (UID: \"ead07c5f-c2fb-42de-9254-059fab1e0ab7\") " pod="kube-system/kas-connection-checker-685895c7c9-qqbzc"
Oct 31 03:07:31.326709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326693    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxk4t\" (UniqueName: \"kubernetes.io/projected/27fe767b-0eac-42e2-a937-e13497bad6bd-kube-api-access-zxk4t\") pod \"network-check-source-6f78d59f99-2zqjk\" (UID: \"27fe767b-0eac-42e2-a937-e13497bad6bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2zqjk"
Oct 31 03:07:31.326761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.326743    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.327257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.327235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw64p\" (UniqueName: \"kubernetes.io/projected/b8f2b00f-643e-412d-9ecb-94f0fd75cf29-kube-api-access-gw64p\") pod \"network-metrics-daemon-s5fjf\" (UID: \"b8f2b00f-643e-412d-9ecb-94f0fd75cf29\") " pod="openshift-multus/network-metrics-daemon-s5fjf"
Oct 31 03:07:31.328081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.328062    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/219ae079-d5b4-4aed-bf97-4766b3a2325d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.329195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.329181    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cni-binary-copy\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.330980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.330964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc279cb7-148d-4787-983c-ea212b73a3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.331095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.331083    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-daemon-config\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.332998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.332978    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46bj\" (UniqueName: \"kubernetes.io/projected/ccfbbf60-9b29-45f3-be5a-0f1282beb2e9-kube-api-access-b46bj\") pod \"migrator-66df5d496b-4tsxv\" (UID: \"ccfbbf60-9b29-45f3-be5a-0f1282beb2e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4tsxv"
Oct 31 03:07:31.333080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333064    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5596k\" (UniqueName: \"kubernetes.io/projected/71bad7a7-3b59-4e50-96d9-59dcd2ec790c-kube-api-access-5596k\") pod \"kas-connection-checker-685895c7c9-jjkm4\" (UID: \"71bad7a7-3b59-4e50-96d9-59dcd2ec790c\") " pod="kube-system/kas-connection-checker-685895c7c9-jjkm4"
Oct 31 03:07:31.333435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333410    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxz8t\" (UniqueName: \"kubernetes.io/projected/88d0ddb1-2f0c-451f-9909-846980b61b9f-kube-api-access-jxz8t\") pod \"ingress-canary-ksv4j\" (UID: \"88d0ddb1-2f0c-451f-9909-846980b61b9f\") " pod="openshift-ingress-canary/ingress-canary-ksv4j"
Oct 31 03:07:31.333569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4cb\" (UniqueName: \"kubernetes.io/projected/fde8fbc1-b8e0-49a5-9725-4586869dce6c-kube-api-access-dl4cb\") pod \"insights-operator-7c4986d449-ldnrv\" (UID: \"fde8fbc1-b8e0-49a5-9725-4586869dce6c\") " pod="openshift-insights/insights-operator-7c4986d449-ldnrv"
Oct 31 03:07:31.333639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333598    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzj8\" (UniqueName: \"kubernetes.io/projected/0381e7c6-4dc6-4a42-a0cf-276261adcb26-kube-api-access-4lzj8\") pod \"service-ca-operator-544d749c8c-jnstn\" (UID: \"0381e7c6-4dc6-4a42-a0cf-276261adcb26\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn"
Oct 31 03:07:31.333772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333756    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.333806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333781    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxfp\" (UniqueName: \"kubernetes.io/projected/801996f5-e5f7-4ef2-93b0-ce1b36c94343-kube-api-access-7qxfp\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.333886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.333872    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wd6\" (UniqueName: \"kubernetes.io/projected/78c7f2a8-6907-4ca4-b636-e0807b250ef9-kube-api-access-j2wd6\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.334876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.334853    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"periodic-gathering-8xxvw-kh2n2\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") " pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:31.334876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.334865    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxt8k\" (UniqueName: \"kubernetes.io/projected/1b558d27-fb04-4431-88a3-22d27045dacb-kube-api-access-qxt8k\") pod \"console-operator-7cd56b8dbc-tkdn9\" (UID: \"1b558d27-fb04-4431-88a3-22d27045dacb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:31.335014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.334928    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv7s\" (UniqueName: \"kubernetes.io/projected/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-kube-api-access-vkv7s\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.335014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.334976    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nqz\" (UniqueName: \"kubernetes.io/projected/dc279cb7-148d-4787-983c-ea212b73a3cc-kube-api-access-m7nqz\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.335106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.335057    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drl7\" (UniqueName: \"kubernetes.io/projected/37a46e66-bcbb-4bc9-b57b-deb78bc76adf-kube-api-access-5drl7\") pod \"network-check-target-2nrm8\" (UID: \"37a46e66-bcbb-4bc9-b57b-deb78bc76adf\") " pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:07:31.335594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.335575    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtm42\" (UniqueName: \"kubernetes.io/projected/dc8ddf63-307d-4908-bff6-df337941dfaf-kube-api-access-gtm42\") pod \"volume-data-source-validator-58c9f5f46d-xc87v\" (UID: \"dc8ddf63-307d-4908-bff6-df337941dfaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xc87v"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.335928    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xxs\" (UniqueName: \"kubernetes.io/projected/c7cf6d1f-3302-4a26-85c1-5a6a8a06f972-kube-api-access-92xxs\") pod \"dns-default-tbchd\" (UID: \"c7cf6d1f-3302-4a26-85c1-5a6a8a06f972\") " pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.335999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zps7s\" (UniqueName: \"kubernetes.io/projected/ec33cd8c-1d9c-437b-8742-661d57128951-kube-api-access-zps7s\") pod \"cluster-monitoring-operator-884576c57-xv98j\" (UID: \"ec33cd8c-1d9c-437b-8742-661d57128951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xv98j"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336014    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtrq\" (UniqueName: \"kubernetes.io/projected/da80e5b0-536a-46a8-aba6-c89002fa216d-kube-api-access-trtrq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2d286\" (UID: \"da80e5b0-536a-46a8-aba6-c89002fa216d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336185    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g4k\" (UniqueName: \"kubernetes.io/projected/b5bfd162-e42e-423a-907c-1ae68fa18ec7-kube-api-access-49g4k\") pod \"migrator-66df5d496b-l4gzk\" (UID: \"b5bfd162-e42e-423a-907c-1ae68fa18ec7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l4gzk"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336260    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336262    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpgl\" (UniqueName: \"kubernetes.io/projected/219ae079-d5b4-4aed-bf97-4766b3a2325d-kube-api-access-fdpgl\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336371    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtvl\" (UniqueName: \"kubernetes.io/projected/d9b1e5a2-6307-4472-9502-5e97c55ac76f-kube-api-access-5xtvl\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.336413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.336368    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmhz\" (UniqueName: \"kubernetes.io/projected/ff779f60-ee44-48e3-b1fa-777374b9788b-kube-api-access-gpmhz\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.338174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.338153    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97wg\" (UniqueName: \"kubernetes.io/projected/4f85eaf9-d372-4e12-8016-0ade17b5badc-kube-api-access-c97wg\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.383092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383068    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383097    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383115    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383134    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383160    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383184    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383191    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-systemd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383210    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-multus\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383220    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383234    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-ovn\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383246    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-run-netns\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383276    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383273    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-systemd\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383307    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-kubelet\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383327    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383350    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383367    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383373    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-socket-dir-parent\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383383    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383406    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.383417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383405    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383437    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383435    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-etc-kubernetes\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383452    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383478    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383487    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d9b1e5a2-6307-4472-9502-5e97c55ac76f-hosts-file\") pod \"node-resolver-mlv4n\" (UID: \"d9b1e5a2-6307-4472-9502-5e97c55ac76f\") " pod="openshift-dns/node-resolver-mlv4n"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383489    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383512    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383517    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-device-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383558    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383563    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-systemd-units\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383569    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-modprobe-d\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383594    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383608    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383612    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383623    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysconfig\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383628    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/801996f5-e5f7-4ef2-93b0-ce1b36c94343-host\") pod \"node-ca-9dj5j\" (UID: \"801996f5-e5f7-4ef2-93b0-ce1b36c94343\") " pod="openshift-image-registry/node-ca-9dj5j"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383629    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383665    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-kubelet-config\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383661    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5feec31a-0c22-428a-9d9a-3c16b3cd6097-dbus\") pod \"global-pull-secret-syncer-bj5qd\" (UID: \"5feec31a-0c22-428a-9d9a-3c16b3cd6097\") " pod="kube-system/global-pull-secret-syncer-bj5qd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383692    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383724    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383743    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-netns\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383752    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383776    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-socket-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383775    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383803    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383816    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-var-lib-kubelet\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383838    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-k8s-cni-cncf-io\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383778    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-etc-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383857    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383866    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-registration-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383896    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383921    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383926    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383948    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-log-socket\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383958    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383983    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383992    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.383999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-bin\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384010    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384022    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-slash\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.384049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384058    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384088    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384096    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc279cb7-148d-4787-983c-ea212b73a3cc-cnibin\") pod \"multus-additional-cni-plugins-m9b2j\" (UID: \"dc279cb7-148d-4787-983c-ea212b73a3cc\") " pod="openshift-multus/multus-additional-cni-plugins-m9b2j"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384115    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384128    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-host\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384139    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384154    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384163    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384179    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-multus-conf-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384196    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-run\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384211    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-cnibin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384210    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384230    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-kubernetes\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384234    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384250    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384264    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384286    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384300    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384315    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-var-lib-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384322    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384331    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-sys\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384332    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-etc-sysctl-conf\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384345    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-kubelet\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384352    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384364    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-node-log\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384384    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-hostroot\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384386    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384403    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-host-cni-netd\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384384    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-var-lib-cni-bin\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384417    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384439    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384463    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384471    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/219ae079-d5b4-4aed-bf97-4766b3a2325d-crio-socket\") pod \"insights-runtime-extractor-svcxz\" (UID: \"219ae079-d5b4-4aed-bf97-4766b3a2325d\") " pod="openshift-insights/insights-runtime-extractor-svcxz"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384488    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384510    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384542    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff779f60-ee44-48e3-b1fa-777374b9788b-lib-modules\") pod \"tuned-c72n4\" (UID: \"ff779f60-ee44-48e3-b1fa-777374b9788b\") " pod="openshift-cluster-node-tuning-operator/tuned-c72n4"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384559    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384577    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb14716-1ea1-40f5-9ac7-7476c9daa35b-host-slash\") pod \"iptables-alerter-brjlb\" (UID: \"5fb14716-1ea1-40f5-9ac7-7476c9daa35b\") " pod="openshift-network-operator/iptables-alerter-brjlb"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384588    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8dcdbcd-d8e5-4400-880b-c89f3362af1b-run-openvswitch\") pod \"ovnkube-node-w9hwd\" (UID: \"f8dcdbcd-d8e5-4400-880b-c89f3362af1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384593    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f85eaf9-d372-4e12-8016-0ade17b5badc-sys-fs\") pod \"aws-ebs-csi-driver-node-m2fmd\" (UID: \"4f85eaf9-d372-4e12-8016-0ade17b5badc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2fmd"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384601    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-system-cni-dir\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.385408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.384619    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/78c7f2a8-6907-4ca4-b636-e0807b250ef9-host-run-multus-certs\") pod \"multus-r8762\" (UID: \"78c7f2a8-6907-4ca4-b636-e0807b250ef9\") " pod="openshift-multus/multus-r8762"
Oct 31 03:07:31.472641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.472620    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:07:31.746449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.746419    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mvwqp"]
Oct 31 03:07:31.750036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.750017    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.792174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.792138    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.792332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.792183    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.792332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.792254    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.792332 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792281    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:31.792332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.792284    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:31.792332 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792320    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792352    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:32.792334777 +0000 UTC m=+2.806896465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792372    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:32.792363447 +0000 UTC m=+2.806925132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792375    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792385    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792402    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792443    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:32.792432124 +0000 UTC m=+2.806993811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:31.792572 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:31.792457    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:32.792451597 +0000 UTC m=+2.807013283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:31.798200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.798178    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:07:31.798455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.798434    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74vl\""
Oct 31 03:07:31.798852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.798823    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:07:31.798950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.798830    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:07:31.799044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.799027    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:07:31.893541 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893498    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-metrics-client-ca\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893550    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-textfile\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893638    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-wtmp\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893668    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-tls\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893799    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893834    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlb4\" (UniqueName: \"kubernetes.io/projected/79933f85-942a-45b0-86f2-c22a0ea57599-kube-api-access-7jlb4\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.893893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893861    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-root\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.894047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.893966    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.894098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.894030    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-sys\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995313    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-metrics-client-ca\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995349    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-textfile\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995388    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-wtmp\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995419    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-tls\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995453    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995476    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlb4\" (UniqueName: \"kubernetes.io/projected/79933f85-942a-45b0-86f2-c22a0ea57599-kube-api-access-7jlb4\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995506    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-root\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995554    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995614    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-sys\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995628    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-wtmp\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995674    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-root\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995811    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79933f85-942a-45b0-86f2-c22a0ea57599-sys\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.995969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995845    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-textfile\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.996162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.995992    5831 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:07:31.996162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.996103    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-metrics-client-ca\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.996395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.996369    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.998300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.998249    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-tls\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:31.998407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:31.998350    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79933f85-942a-45b0-86f2-c22a0ea57599-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:32.048829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.048796    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlb4\" (UniqueName: \"kubernetes.io/projected/79933f85-942a-45b0-86f2-c22a0ea57599-kube-api-access-7jlb4\") pod \"node-exporter-mvwqp\" (UID: \"79933f85-942a-45b0-86f2-c22a0ea57599\") " pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:32.065823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.065793    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvwqp"
Oct 31 03:07:32.078452 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:32.078426    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79933f85_942a_45b0_86f2_c22a0ea57599.slice/crio-c46a576b31892a181555ab6db21493b023fb762248cec6efcaebbf3b6891bc35 WatchSource:0}: Error finding container c46a576b31892a181555ab6db21493b023fb762248cec6efcaebbf3b6891bc35: Status 404 returned error can't find the container with id c46a576b31892a181555ab6db21493b023fb762248cec6efcaebbf3b6891bc35
Oct 31 03:07:32.212550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.212483    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvwqp" event={"ID":"79933f85-942a-45b0-86f2-c22a0ea57599","Type":"ContainerStarted","Data":"c46a576b31892a181555ab6db21493b023fb762248cec6efcaebbf3b6891bc35"}
Oct 31 03:07:32.214090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.214059    5831 generic.go:356] "Generic (PLEG): container finished" podID="fde8fbc1-b8e0-49a5-9725-4586869dce6c" containerID="115a18fb41cbd6bbc355739534cc4748d397d068ccc3f28728a0a11dd9781005" exitCode=0
Oct 31 03:07:32.214211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.214133    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ldnrv" event={"ID":"fde8fbc1-b8e0-49a5-9725-4586869dce6c","Type":"ContainerDied","Data":"115a18fb41cbd6bbc355739534cc4748d397d068ccc3f28728a0a11dd9781005"}
Oct 31 03:07:32.214616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.214598    5831 scope.go:122] "RemoveContainer" containerID="115a18fb41cbd6bbc355739534cc4748d397d068ccc3f28728a0a11dd9781005"
Oct 31 03:07:32.442889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.442865    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:32.556485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.556414    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6489c4dbb7-6rbbc"
Oct 31 03:07:32.803197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.803170    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.803213    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.803251    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:32.803292    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803347    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803357    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:32.803371 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803383    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803408    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803352    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803428    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:34.803406693 +0000 UTC m=+4.817968393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803450    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:34.803440051 +0000 UTC m=+4.818001743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803466    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:34.803456043 +0000 UTC m=+4.818017731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:32.803655 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:32.803495    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:34.803474006 +0000 UTC m=+4.818035721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:33.220337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.220298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ldnrv" event={"ID":"fde8fbc1-b8e0-49a5-9725-4586869dce6c","Type":"ContainerStarted","Data":"402d130a0e312f9f8cab436f4448ce7e079c12947fce3bbcec965eafac9f8fe2"}
Oct 31 03:07:33.221959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.221933    5831 generic.go:356] "Generic (PLEG): container finished" podID="0381e7c6-4dc6-4a42-a0cf-276261adcb26" containerID="c251e0c29bf74642b9e91df0a8891d4ffe10c30cf0f15cfa2d0f2f69486310b0" exitCode=0
Oct 31 03:07:33.222084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.221987    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn" event={"ID":"0381e7c6-4dc6-4a42-a0cf-276261adcb26","Type":"ContainerDied","Data":"c251e0c29bf74642b9e91df0a8891d4ffe10c30cf0f15cfa2d0f2f69486310b0"}
Oct 31 03:07:33.222268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.222254    5831 scope.go:122] "RemoveContainer" containerID="c251e0c29bf74642b9e91df0a8891d4ffe10c30cf0f15cfa2d0f2f69486310b0"
Oct 31 03:07:33.224089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.224070    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf" event={"ID":"009cba45-9f0a-4991-8ab1-86b7ed158f79","Type":"ContainerStarted","Data":"2a737b5e53394b2b974a93c4fdf107bfd7c1efe7b709c4ae24cfd0931caf6f80"}
Oct 31 03:07:33.224180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.224109    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-26sqf" event={"ID":"009cba45-9f0a-4991-8ab1-86b7ed158f79","Type":"ContainerStarted","Data":"855e21b8d5b1f6688735598e53bdbf15c289e3b94151620616032c0c27be0fd0"}
Oct 31 03:07:33.477276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.477188    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:33.480851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.480828    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tkdn9"
Oct 31 03:07:33.652924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.652895    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wrpcw-4ddhc"]
Oct 31 03:07:33.656150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.656124    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.710403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.710373    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wrpcw-4ddhc"]
Oct 31 03:07:33.814916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.814823    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.814916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.814870    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.815097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.815031    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcp5f\" (UniqueName: \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.815097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.815090    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916053    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916100    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916192    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp5f\" (UniqueName: \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916227    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916413    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916840    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.916984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.916964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:33.969316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:33.969290    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp5f\" (UniqueName: \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\") pod \"periodic-gathering-wrpcw-4ddhc\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") " pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:34.230640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.230605    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jnstn" event={"ID":"0381e7c6-4dc6-4a42-a0cf-276261adcb26","Type":"ContainerStarted","Data":"680c90f2062f267a5ab64d8748e0770dfe1701d0edcb84400a54acede5c874a6"}
Oct 31 03:07:34.232250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.232221    5831 generic.go:356] "Generic (PLEG): container finished" podID="79933f85-942a-45b0-86f2-c22a0ea57599" containerID="71b6e314784e85208eb31ceeb3686f1a4ac45ed25934a6a23f76d67d6efb9c51" exitCode=0
Oct 31 03:07:34.232361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.232274    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvwqp" event={"ID":"79933f85-942a-45b0-86f2-c22a0ea57599","Type":"ContainerDied","Data":"71b6e314784e85208eb31ceeb3686f1a4ac45ed25934a6a23f76d67d6efb9c51"}
Oct 31 03:07:34.269615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.269585    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:07:34.472473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.472444    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tbchd"
Oct 31 03:07:34.645846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.645754    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wrpcw-4ddhc"]
Oct 31 03:07:34.649151 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:07:34.649126    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ebbd46_f187_4f72_b2cc_22ea7dde3ad9.slice/crio-b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44 WatchSource:0}: Error finding container b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44: Status 404 returned error can't find the container with id b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44
Oct 31 03:07:34.826197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.826162    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:34.826366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.826204    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:34.826366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.826254    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:34.826366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:34.826273    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:34.826366 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826327    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:34.826366 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826348    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826367    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826385    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826397    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826400    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:38.826381051 +0000 UTC m=+8.840942755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826419    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:38.826412025 +0000 UTC m=+8.840973714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826432    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:38.826425133 +0000 UTC m=+8.840986818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:34.826520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:34.826444    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:38.826436907 +0000 UTC m=+8.840998594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:35.237696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.237660    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc" event={"ID":"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9","Type":"ContainerStarted","Data":"20ddd08672d924d4a71b2c75cf29ff40db8e11e809b278bc2dcaa982603532f0"}
Oct 31 03:07:35.238200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.237709    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc" event={"ID":"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9","Type":"ContainerStarted","Data":"b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44"}
Oct 31 03:07:35.240138 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.240114    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvwqp" event={"ID":"79933f85-942a-45b0-86f2-c22a0ea57599","Type":"ContainerStarted","Data":"1742c832376815f21f4b16e6c02c368358111ea94056925d9e4501780668054a"}
Oct 31 03:07:35.240278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.240167    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvwqp" event={"ID":"79933f85-942a-45b0-86f2-c22a0ea57599","Type":"ContainerStarted","Data":"1186b6f178908bcbed9ea68410e9b4609d6c0d566f9dcb8d30968ecabb66bdf3"}
Oct 31 03:07:35.343433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.343381    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc" podStartSLOduration=2.343366944 podStartE2EDuration="2.343366944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:35.34282008 +0000 UTC m=+5.357381793" watchObservedRunningTime="2026-10-31 03:07:35.343366944 +0000 UTC m=+5.357928643"
Oct 31 03:07:35.455397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:35.455349    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mvwqp" podStartSLOduration=2.517789466 podStartE2EDuration="4.455334917s" totalImagesPullingTime="1.861055139s" totalInitContainerRuntime="76.490312ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:35.454680186 +0000 UTC m=+5.469241930" watchObservedRunningTime="2026-10-31 03:07:35.455334917 +0000 UTC m=+5.469896622"
Oct 31 03:07:36.245922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:36.245892    5831 generic.go:356] "Generic (PLEG): container finished" podID="da80e5b0-536a-46a8-aba6-c89002fa216d" containerID="0ff2c4155e619cf5ecaeac92c50d2600f8d62dc7c0eb5ec44911e43fa91d2520" exitCode=0
Oct 31 03:07:36.246294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:36.245965    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286" event={"ID":"da80e5b0-536a-46a8-aba6-c89002fa216d","Type":"ContainerDied","Data":"0ff2c4155e619cf5ecaeac92c50d2600f8d62dc7c0eb5ec44911e43fa91d2520"}
Oct 31 03:07:36.246521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:36.246506    5831 scope.go:122] "RemoveContainer" containerID="0ff2c4155e619cf5ecaeac92c50d2600f8d62dc7c0eb5ec44911e43fa91d2520"
Oct 31 03:07:36.258287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:36.258271    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6489c4dbb7-6rbbc_d608bf35-2830-4486-a05f-ac59ddb3c8e6/router/0.log"
Oct 31 03:07:36.508724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:36.508692    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ksv4j_88d0ddb1-2f0c-451f-9909-846980b61b9f/serve-healthcheck-canary/0.log"
Oct 31 03:07:37.252142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:37.252104    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2d286" event={"ID":"da80e5b0-536a-46a8-aba6-c89002fa216d","Type":"ContainerStarted","Data":"3e56f08b980e344a99d7b8ba47b36ca5974d8034ca681838a4fa126809ffbf3a"}
Oct 31 03:07:37.893901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:37.893868    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2nrm8"
Oct 31 03:07:38.866329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:38.866293    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:38.866329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:38.866336    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:38.866390    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:38.866412    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866452    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866476    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866507    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866517    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866548    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:46.866509373 +0000 UTC m=+16.881071061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866567    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866571    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:46.866560811 +0000 UTC m=+16.881122497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866636    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:46.866618127 +0000 UTC m=+16.881179821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:38.866874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:38.866662    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:46.866651173 +0000 UTC m=+16.881212861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:39.991367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:39.991335    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:07:39.991800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:39.991557    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" podUID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" containerName="insights-gathering" containerID="cri-o://ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c" gracePeriod=30
Oct 31 03:07:40.142427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.142393    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:40.142705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.142686    5831 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:40.232682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.232657    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:40.264813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.264734    5831 generic.go:356] "Generic (PLEG): container finished" podID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" containerID="ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c" exitCode=2
Oct 31 03:07:40.264813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.264787    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" event={"ID":"7cb3c50f-ca36-437c-b720-034fa17cb3e1","Type":"ContainerDied","Data":"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"}
Oct 31 03:07:40.264813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.264793    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2"
Oct 31 03:07:40.265042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.264832    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8xxvw-kh2n2" event={"ID":"7cb3c50f-ca36-437c-b720-034fa17cb3e1","Type":"ContainerDied","Data":"92da36f0fdbbda3f40c5aa9effd8f0a3eadd32b438653fb6f2d8c91b7a4ca8d5"}
Oct 31 03:07:40.265042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.264862    5831 scope.go:122] "RemoveContainer" containerID="ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"
Oct 31 03:07:40.274665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.274645    5831 scope.go:122] "RemoveContainer" containerID="ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"
Oct 31 03:07:40.274961 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:40.274944    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c\": container with ID starting with ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c not found: ID does not exist" containerID="ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"
Oct 31 03:07:40.275007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.274971    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c"} err="failed to get container status \"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c\": rpc error: code = NotFound desc = could not find container \"ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c\": container with ID starting with ed665b0703b71fd69570d9f4da933d88f7333ab86f93590044e9e7b1a6bbbb2c not found: ID does not exist"
Oct 31 03:07:40.281555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.281519    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") pod \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") "
Oct 31 03:07:40.281640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.281576    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") pod \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") "
Oct 31 03:07:40.281755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.281652    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") pod \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") "
Oct 31 03:07:40.281808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.281775    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") pod \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\" (UID: \"7cb3c50f-ca36-437c-b720-034fa17cb3e1\") "
Oct 31 03:07:40.281979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.281956    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path" pod "7cb3c50f-ca36-437c-b720-034fa17cb3e1" (UID: "7cb3c50f-ca36-437c-b720-034fa17cb3e1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:07:40.282135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.282117    5831 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7cb3c50f-ca36-437c-b720-034fa17cb3e1-archives-path\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:40.282243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.282217    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle" pod "7cb3c50f-ca36-437c-b720-034fa17cb3e1" (UID: "7cb3c50f-ca36-437c-b720-034fa17cb3e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:40.282345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.282301    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle" pod "7cb3c50f-ca36-437c-b720-034fa17cb3e1" (UID: "7cb3c50f-ca36-437c-b720-034fa17cb3e1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:40.283879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.283855    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl" pod "7cb3c50f-ca36-437c-b720-034fa17cb3e1" (UID: "7cb3c50f-ca36-437c-b720-034fa17cb3e1"). InnerVolumeSpecName "kube-api-access-mpkhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:40.383364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.383333    5831 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-service-ca-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:40.383364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.383359    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkhl\" (UniqueName: \"kubernetes.io/projected/7cb3c50f-ca36-437c-b720-034fa17cb3e1-kube-api-access-mpkhl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:40.383364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.383369    5831 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb3c50f-ca36-437c-b720-034fa17cb3e1-trusted-ca-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:40.676070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.676036    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:07:40.679989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:40.679963    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-8xxvw-kh2n2"]
Oct 31 03:07:41.407286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:41.407250    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9hwd"
Oct 31 03:07:42.133244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:42.133208    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" path="/var/lib/kubelet/pods/7cb3c50f-ca36-437c-b720-034fa17cb3e1/volumes"
Oct 31 03:07:42.977699 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:42.977663    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:07:46.940110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:46.940074    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:46.940110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:46.940114    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:46.940167    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:46.940201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"image-registry-5b48949c57-xszl6\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") " pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940242    5831 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940299    5831 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940304    5831 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940324    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:08:02.940305445 +0000 UTC m=+32.954867152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"installation-pull-secrets" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940328    5831 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b48949c57-xszl6: object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940328    5831 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940351    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:08:02.940338901 +0000 UTC m=+32.954900611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-private-configuration" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940375    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:08:02.940360431 +0000 UTC m=+32.954922138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"image-registry-tls" not registered
Oct 31 03:07:46.940587 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:07:46.940398    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca podName:54b4593e-3511-4f4c-8dc6-b2599c75efe4 nodeName:}" failed. No retries permitted until 2026-10-31 03:08:02.940388126 +0000 UTC m=+32.954949819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca") pod "image-registry-5b48949c57-xszl6" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4") : object "openshift-image-registry"/"trusted-ca" not registered
Oct 31 03:07:54.143338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:54.143301    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhzmz"
Oct 31 03:07:56.203863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.203809    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" podUID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" containerName="registry" containerID="cri-o://14160c639162f339c0c937614dcf097f245d208d8747b47bd4292a3772b4e6a4" gracePeriod=30
Oct 31 03:07:56.337167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.337133    5831 generic.go:356] "Generic (PLEG): container finished" podID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" containerID="14160c639162f339c0c937614dcf097f245d208d8747b47bd4292a3772b4e6a4" exitCode=0
Oct 31 03:07:56.337314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.337205    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" event={"ID":"54b4593e-3511-4f4c-8dc6-b2599c75efe4","Type":"ContainerDied","Data":"14160c639162f339c0c937614dcf097f245d208d8747b47bd4292a3772b4e6a4"}
Oct 31 03:07:56.468416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.468358    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:56.630822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630789    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630867    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630893    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630913    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630957    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.630988    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.631030    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.631072    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") pod \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\" (UID: \"54b4593e-3511-4f4c-8dc6-b2599c75efe4\") "
Oct 31 03:07:56.631597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.631560    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:56.632223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.632196    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:07:56.634325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.634298    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:56.634617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.634558    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:56.635284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.635262    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:07:56.635599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.635559    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "kube-api-access-fkp8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:07:56.635708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.635682    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:07:56.640917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.640885    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted" pod "54b4593e-3511-4f4c-8dc6-b2599c75efe4" (UID: "54b4593e-3511-4f4c-8dc6-b2599c75efe4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732053    5831 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-trusted-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732078    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkp8k\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-kube-api-access-fkp8k\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732091    5831 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732100    5831 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54b4593e-3511-4f4c-8dc6-b2599c75efe4-registry-certificates\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732113    5831 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54b4593e-3511-4f4c-8dc6-b2599c75efe4-ca-trust-extracted\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732123    5831 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-image-registry-private-configuration\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732132    5831 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54b4593e-3511-4f4c-8dc6-b2599c75efe4-installation-pull-secrets\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:56.732135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:56.732140    5831 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54b4593e-3511-4f4c-8dc6-b2599c75efe4-bound-sa-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:07:57.342250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:57.342219    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b48949c57-xszl6"
Oct 31 03:07:57.342692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:57.342213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b48949c57-xszl6" event={"ID":"54b4593e-3511-4f4c-8dc6-b2599c75efe4","Type":"ContainerDied","Data":"cc2d1dde4ea5d9cc3283948d2bc16977247209fbb70eb89eda268e250216098e"}
Oct 31 03:07:57.342692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:57.342356    5831 scope.go:122] "RemoveContainer" containerID="14160c639162f339c0c937614dcf097f245d208d8747b47bd4292a3772b4e6a4"
Oct 31 03:07:57.541093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:57.541061    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b48949c57-xszl6"]
Oct 31 03:07:57.587235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:57.587180    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b48949c57-xszl6"]
Oct 31 03:07:58.132313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:07:58.132282    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" path="/var/lib/kubelet/pods/54b4593e-3511-4f4c-8dc6-b2599c75efe4/volumes"
Oct 31 03:08:12.977361 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:08:12.977324    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:08:42.976614 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:08:42.976563    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:09:01.573402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:01.573365    5831 generic.go:356] "Generic (PLEG): container finished" podID="44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" containerID="20ddd08672d924d4a71b2c75cf29ff40db8e11e809b278bc2dcaa982603532f0" exitCode=0
Oct 31 03:09:01.573861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:01.573438    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc" event={"ID":"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9","Type":"ContainerDied","Data":"20ddd08672d924d4a71b2c75cf29ff40db8e11e809b278bc2dcaa982603532f0"}
Oct 31 03:09:02.726343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.726320    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:09:02.865229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865150    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\") pod \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") "
Oct 31 03:09:02.865229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865185    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\") pod \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") "
Oct 31 03:09:02.865229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865208    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\" (UniqueName: \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\") pod \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") "
Oct 31 03:09:02.865447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865239    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\") pod \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\" (UID: \"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9\") "
Oct 31 03:09:02.865571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865487    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path" pod "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" (UID: "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:09:02.865631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865609    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle" pod "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" (UID: "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:09:02.865675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.865654    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle" pod "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" (UID: "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:09:02.867510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.867488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f" pod "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" (UID: "44ebbd46-f187-4f72-b2cc-22ea7dde3ad9"). InnerVolumeSpecName "kube-api-access-kcp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:09:02.965803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.965770    5831 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-archives-path\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:09:02.965803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.965798    5831 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-service-ca-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:09:02.965803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.965808    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcp5f\" (UniqueName: \"kubernetes.io/projected/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-kube-api-access-kcp5f\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:09:02.966037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:02.965819    5831 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44ebbd46-f187-4f72-b2cc-22ea7dde3ad9-trusted-ca-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:09:03.582215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:03.582185    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc"
Oct 31 03:09:03.582392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:03.582189    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wrpcw-4ddhc" event={"ID":"44ebbd46-f187-4f72-b2cc-22ea7dde3ad9","Type":"ContainerDied","Data":"b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44"}
Oct 31 03:09:03.582392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:09:03.582325    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40ec656c8dc0e44514de56a151246e964d3ca40e44049a0909391527a8e0f44"
Oct 31 03:09:12.976361 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:09:12.976316    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:09:42.977228 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:09:42.977189    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:10:10.916891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.916858    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"]
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917277    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" containerName="insights-gathering"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917294    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" containerName="insights-gathering"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917318    5831 cpu_manager.go:414] "containerMap: removing container" podUID="44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" containerName="insights-gathering"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917325    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" containerName="insights-gathering"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917342    5831 cpu_manager.go:414] "containerMap: removing container" podUID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" containerName="registry"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917350    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" containerName="registry"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917438    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b4593e-3511-4f4c-8dc6-b2599c75efe4" containerName="registry"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917455    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb3c50f-ca36-437c-b720-034fa17cb3e1" containerName="insights-gathering"
Oct 31 03:10:10.917512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.917466    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ebbd46-f187-4f72-b2cc-22ea7dde3ad9" containerName="insights-gathering"
Oct 31 03:10:10.920861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.920839    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:10.971231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.971188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:10.971231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.971231    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:10.971457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:10.971253    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvzc\" (UniqueName: \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.023402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.023360    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4wskf\""
Oct 31 03:10:11.023402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.023372    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 03:10:11.023662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.023446    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 03:10:11.024302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.024283    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"]
Oct 31 03:10:11.053390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.053363    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"]
Oct 31 03:10:11.056792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.056775    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.071801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.071777    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.071938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.071807    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.071938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.071831    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvzc\" (UniqueName: \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.072116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.072097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.072188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.072158    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.105386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.105351    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dd6r7\""
Oct 31 03:10:11.110918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.110893    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"]
Oct 31 03:10:11.125588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.125561    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvzc\" (UniqueName: \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.143663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.143635    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"]
Oct 31 03:10:11.147086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.147067    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.172833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.172756    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.172833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.172797    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmql\" (UniqueName: \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.173029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.172926    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.206623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.206597    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-j9kkt\""
Oct 31 03:10:11.206883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.206862    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"]
Oct 31 03:10:11.237520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.237485    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:11.273653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273623    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.273855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273680    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.273855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273751    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.273855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmql\" (UniqueName: \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.273855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273831    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbclt\" (UniqueName: \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.274038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.273865    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.274095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.274078    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.274141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.274121    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.328540 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.328483    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmql\" (UniqueName: \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.372601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.372565    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:11.374882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.374858    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.375105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.374903    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbclt\" (UniqueName: \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.375105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.374937    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.375287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.375267    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.375322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.375283    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.430662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.430631    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbclt\" (UniqueName: \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.462626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.462598    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:11.614308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.614276    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"]
Oct 31 03:10:11.618156 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:11.618114    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad51f1e_3f7c_4593_8bb1_2e3e5e18396a.slice/crio-a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72 WatchSource:0}: Error finding container a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72: Status 404 returned error can't find the container with id a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72
Oct 31 03:10:11.747374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.747345    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"]
Oct 31 03:10:11.750664 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:11.750637    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3139e21_cc0c_4910_8a21_e99bc0e5662e.slice/crio-2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598 WatchSource:0}: Error finding container 2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598: Status 404 returned error can't find the container with id 2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598
Oct 31 03:10:11.828279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.828235    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerStarted","Data":"2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598"}
Oct 31 03:10:11.828905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.828869    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"]
Oct 31 03:10:11.829752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:11.829726    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh" event={"ID":"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a","Type":"ContainerStarted","Data":"a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72"}
Oct 31 03:10:11.832843 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:11.832812    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2645bc_3f45_46bf_9998_bc27374c789f.slice/crio-fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3 WatchSource:0}: Error finding container fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3: Status 404 returned error can't find the container with id fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3
Oct 31 03:10:12.834574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:12.834521    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc" event={"ID":"5d2645bc-3f45-46bf-9998-bc27374c789f","Type":"ContainerStarted","Data":"fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3"}
Oct 31 03:10:12.987010 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:10:12.986970    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:10:13.742714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.742671    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"]
Oct 31 03:10:13.746736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.746713    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.795097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.795055    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.795097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.795103    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.795382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.795163    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmq8n\" (UniqueName: \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.841590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.841558    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5vghm\""
Oct 31 03:10:13.842929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.842910    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"]
Oct 31 03:10:13.843710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.843680    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"]
Oct 31 03:10:13.848091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.848069    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.896264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896229    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.896562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896284    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.896562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896320    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.896562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896385    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmq8n\" (UniqueName: \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.896562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896474    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.896562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896508    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxhn\" (UniqueName: \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.896920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896714    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.896920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.896766    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.899083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.899016    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pjnqp\""
Oct 31 03:10:13.910402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.910359    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"]
Oct 31 03:10:13.946885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.946851    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"]
Oct 31 03:10:13.950844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.950816    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:13.954279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.954239    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmq8n\" (UniqueName: \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:13.997713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997605    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxhn\" (UniqueName: \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.997867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997713    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.997867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997782    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:13.997867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997819    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfpl\" (UniqueName: \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:13.997867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997838    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.997867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.997855    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:13.998295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.998271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.998295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.998245    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:13.999389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:13.999369    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-4mrtn\""
Oct 31 03:10:14.003730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.003641    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"]
Oct 31 03:10:14.048244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.046284    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"]
Oct 31 03:10:14.052336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.051784    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.053202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.053175    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxhn\" (UniqueName: \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:14.065710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.065678    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:14.099388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099349    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.099555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099412    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dwr\" (UniqueName: \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.099555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099511    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.099693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099576    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfpl\" (UniqueName: \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.099693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099640    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.099795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.099712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.100601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.100554    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.100918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.100863    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.102034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.101930    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2796r\""
Oct 31 03:10:14.103927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.103850    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"]
Oct 31 03:10:14.159808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.159781    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfpl\" (UniqueName: \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.168007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.167828    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:14.200797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.200767    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dwr\" (UniqueName: \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.200962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.200808    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.200962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.200881    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.201311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.201289    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.201960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.201905    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.261855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.261788    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dwr\" (UniqueName: \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.281607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.281356    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:14.382184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.382153    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:14.462779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.462696    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"]
Oct 31 03:10:14.476104 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:14.476077    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4588f57_e259_4eef_9188_2cf6c8321f22.slice/crio-7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2 WatchSource:0}: Error finding container 7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2: Status 404 returned error can't find the container with id 7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2
Oct 31 03:10:14.570802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.570699    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"]
Oct 31 03:10:14.576048 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:14.576014    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae1928b_abe2_4033_868b_a126462851db.slice/crio-304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c WatchSource:0}: Error finding container 304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c: Status 404 returned error can't find the container with id 304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c
Oct 31 03:10:14.655776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.655745    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"]
Oct 31 03:10:14.661208 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:14.661177    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce43062_cb66_4466_8a10_f7c022b16588.slice/crio-d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e WatchSource:0}: Error finding container d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e: Status 404 returned error can't find the container with id d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e
Oct 31 03:10:14.785199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.785163    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"]
Oct 31 03:10:14.792406 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:14.792369    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3848a046_c893_4d98_a779_80b6d84f52ed.slice/crio-24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f WatchSource:0}: Error finding container 24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f: Status 404 returned error can't find the container with id 24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f
Oct 31 03:10:14.846647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.846550    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerStarted","Data":"7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2"}
Oct 31 03:10:14.848469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.848397    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6" event={"ID":"9ce43062-cb66-4466-8a10-f7c022b16588","Type":"ContainerStarted","Data":"d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e"}
Oct 31 03:10:14.850210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.850181    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerStarted","Data":"24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f"}
Oct 31 03:10:14.851841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:14.851813    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv" event={"ID":"8ae1928b-abe2-4033-868b-a126462851db","Type":"ContainerStarted","Data":"304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c"}
Oct 31 03:10:15.977738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:15.977700    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"]
Oct 31 03:10:15.981693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:15.981664    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.079907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.079865    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"]
Oct 31 03:10:16.119145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.119105    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.119320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.119189    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkqf\" (UniqueName: \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.119320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.119247    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.220090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.220043    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.220295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.220157    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkqf\" (UniqueName: \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.220798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.220771    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.221803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.221777    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.222284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.222261    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.282034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.281960    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkqf\" (UniqueName: \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.305776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.305422    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:16.368237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.366929    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"]
Oct 31 03:10:16.375961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.375371    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.423635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.422519    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.423635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.423336    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5zj\" (UniqueName: \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.423635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.423382    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.424649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.424626    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"]
Oct 31 03:10:16.524321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.524284    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.524519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.524343    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5zj\" (UniqueName: \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.524519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.524389    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.524871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.524848    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.525039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.525024    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.602997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.590007    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5zj\" (UniqueName: \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.700688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.700341    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:16.743214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.743183    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"]
Oct 31 03:10:16.749516 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:16.749177    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb0341d_6afa_46f2_8129_ec3c694a95a5.slice/crio-feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb WatchSource:0}: Error finding container feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb: Status 404 returned error can't find the container with id feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb
Oct 31 03:10:16.873935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:16.873844    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2" event={"ID":"4cb0341d-6afa-46f2-8129-ec3c694a95a5","Type":"ContainerStarted","Data":"feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb"}
Oct 31 03:10:17.096001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.095936    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"]
Oct 31 03:10:17.451021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.450988    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:10:17.456500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.456455    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.505869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.505832    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:10:17.523799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.523767    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 03:10:17.532289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532254    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2tv\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532317    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532388    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532414    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532453    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532508    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532556    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532668    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.532870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.532699    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.557739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.557704    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:10:17.633229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633184    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2tv\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633245    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633301    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633326    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633391    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633424    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633474    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.633839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633500    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.634000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.633962    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.635746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.635707    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.636254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.635884    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.636254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.636047    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.636254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.636189    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.638200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.638170    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.643862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.643835    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.690980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.690944    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2tv\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.692101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.692066    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-22gdr\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.761050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.760958    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"]
Oct 31 03:10:17.765830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.765802    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.775711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.775687    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:17.822114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.822080    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"]
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.835854    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.835904    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdjl\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-kube-api-access-tpdjl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.835941    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.835978    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.836029    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.836107    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.836135    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.836172    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.836309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.836243    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.892672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.885006    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-2tgfn\""
Oct 31 03:10:17.903573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.903517    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerStarted","Data":"87c0dc4d152a8f8ddfbd77bca924a2d861bc3a89bac8afb272e1563bbaea37ae"}
Oct 31 03:10:17.937456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937411    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937482    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937555    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937589    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937630    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdjl\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-kube-api-access-tpdjl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937672    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937711    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937751    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.937908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.937846    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.938690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.938663    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.938977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.938956    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.939078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.939032    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.939268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.939246    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.939742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.939722    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.942585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.942565    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:17.950419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:17.950387    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:18.002688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.002619    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdjl\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-kube-api-access-tpdjl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:18.004903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.004808    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph\" (UID: \"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:18.083647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.083267    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:18.238131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.238043    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:10:18.525675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.525623    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"]
Oct 31 03:10:18.924349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.924251    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph" event={"ID":"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3","Type":"ContainerStarted","Data":"3985832892eb76115ed46d2abbfa21d037728006b666928d32592ee7fdf2a909"}
Oct 31 03:10:18.928027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:18.927996    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerStarted","Data":"970fa87d21f2efb8fd106e87a54243a0f3145829b20b17635b5ceab474aab8a4"}
Oct 31 03:10:29.596167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:29.595247    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:29.596167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:29.595336    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:29.596167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:29.595380    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:30.060882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.060801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerStarted","Data":"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"}
Oct 31 03:10:30.776874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.776839    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:30.783719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.783691    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:30.851368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.851326    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:30.851477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.851431    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:30.851524 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.851477    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:10:30.895151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:30.895084    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podStartSLOduration=2.553766026 podStartE2EDuration="13.894090812s" totalImagesPullingTime="11.340324786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:30.251148645 +0000 UTC m=+180.265710354" watchObservedRunningTime="2026-10-31 03:10:30.894090812 +0000 UTC m=+180.908652523"
Oct 31 03:10:31.073292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.073253    5831 generic.go:356] "Generic (PLEG): container finished" podID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerID="08e9de5478f360588629896459ef14c221d61419e3958785a2253f8503d45f6b" exitCode=0
Oct 31 03:10:31.073501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.073345    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerDied","Data":"08e9de5478f360588629896459ef14c221d61419e3958785a2253f8503d45f6b"}
Oct 31 03:10:31.076465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.076438    5831 generic.go:356] "Generic (PLEG): container finished" podID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerID="6e4a45a0855aded2de564c1d9c484e76ba56dabd95fece39993656ee9389c58b" exitCode=0
Oct 31 03:10:31.076630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.076518    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerDied","Data":"6e4a45a0855aded2de564c1d9c484e76ba56dabd95fece39993656ee9389c58b"}
Oct 31 03:10:31.080008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.079978    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerID="3e6999f0db67ab7579169448cc72aa75e890bbccfa73e4b48136f2de1d5c21ab" exitCode=0
Oct 31 03:10:31.080115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.080070    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh" event={"ID":"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a","Type":"ContainerDied","Data":"3e6999f0db67ab7579169448cc72aa75e890bbccfa73e4b48136f2de1d5c21ab"}
Oct 31 03:10:31.084454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.084384    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph" event={"ID":"c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3","Type":"ContainerStarted","Data":"15c9f4d6e7f2c726c2b07a07a0d29f650197760a2495e41016daba6a05bd7c83"}
Oct 31 03:10:31.094146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.094110    5831 generic.go:356] "Generic (PLEG): container finished" podID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerID="711c2519a836d75f6db8eb9eb5a21bf3bc1050835182ab89ebe486074aa727eb" exitCode=0
Oct 31 03:10:31.094650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.094619    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc" event={"ID":"5d2645bc-3f45-46bf-9998-bc27374c789f","Type":"ContainerDied","Data":"711c2519a836d75f6db8eb9eb5a21bf3bc1050835182ab89ebe486074aa727eb"}
Oct 31 03:10:31.094929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.094908    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:31.096331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.096241    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:10:31.738452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:31.738394    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph" podStartSLOduration=2.423007498 podStartE2EDuration="14.738378737s" totalImagesPullingTime="12.315371239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:31.737873083 +0000 UTC m=+181.752434790" watchObservedRunningTime="2026-10-31 03:10:31.738378737 +0000 UTC m=+181.752940442"
Oct 31 03:10:32.084033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:32.083949    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:32.089689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:32.089656    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:32.100959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:32.100898    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:32.101501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:32.101471    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph"
Oct 31 03:10:33.107617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:33.107582    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerStarted","Data":"569661da4daf460082e9f472c82717b8adefd3a39c4eb5912180e68d02d7a8be"}
Oct 31 03:10:33.112271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:33.112234    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerStarted","Data":"7c91a06f772514e5898e9d8303dd9784d726d0ecdbb382018af5450d6ff91e7c"}
Oct 31 03:10:34.121054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.120914    5831 generic.go:356] "Generic (PLEG): container finished" podID="3848a046-c893-4d98-a779-80b6d84f52ed" containerID="28aa3e98b525ecb2e868de888c14f566bfdc9f2b8ed558709467641019ad9bd7" exitCode=0
Oct 31 03:10:34.121054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.121000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerDied","Data":"28aa3e98b525ecb2e868de888c14f566bfdc9f2b8ed558709467641019ad9bd7"}
Oct 31 03:10:34.125271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.125242    5831 generic.go:356] "Generic (PLEG): container finished" podID="8ae1928b-abe2-4033-868b-a126462851db" containerID="f900102ad1075f7fbc409e0a08d3e67b6356c8bb19bd8a1152a132733412865b" exitCode=0
Oct 31 03:10:34.125963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.125936    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv" event={"ID":"8ae1928b-abe2-4033-868b-a126462851db","Type":"ContainerDied","Data":"f900102ad1075f7fbc409e0a08d3e67b6356c8bb19bd8a1152a132733412865b"}
Oct 31 03:10:34.128717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.128597    5831 generic.go:356] "Generic (PLEG): container finished" podID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerID="1bc864fe49833ed6d931d9eefc3c09a5b5f3e2f816424a967fa396eb12fe2245" exitCode=0
Oct 31 03:10:34.132027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.132005    5831 generic.go:356] "Generic (PLEG): container finished" podID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerID="569661da4daf460082e9f472c82717b8adefd3a39c4eb5912180e68d02d7a8be" exitCode=0
Oct 31 03:10:34.134698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.134675    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerDied","Data":"1bc864fe49833ed6d931d9eefc3c09a5b5f3e2f816424a967fa396eb12fe2245"}
Oct 31 03:10:34.134791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.134723    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerDied","Data":"569661da4daf460082e9f472c82717b8adefd3a39c4eb5912180e68d02d7a8be"}
Oct 31 03:10:34.136389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.135256    5831 generic.go:356] "Generic (PLEG): container finished" podID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerID="7c91a06f772514e5898e9d8303dd9784d726d0ecdbb382018af5450d6ff91e7c" exitCode=0
Oct 31 03:10:34.136389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:34.136339    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerDied","Data":"7c91a06f772514e5898e9d8303dd9784d726d0ecdbb382018af5450d6ff91e7c"}
Oct 31 03:10:35.143426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:35.143383    5831 generic.go:356] "Generic (PLEG): container finished" podID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerID="029ab3fd2cd0f95428e0c29ba8d7ca533e02f8039a365bb0c1b6037981ffae04" exitCode=0
Oct 31 03:10:35.144022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:35.143478    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2" event={"ID":"4cb0341d-6afa-46f2-8129-ec3c694a95a5","Type":"ContainerDied","Data":"029ab3fd2cd0f95428e0c29ba8d7ca533e02f8039a365bb0c1b6037981ffae04"}
Oct 31 03:10:35.145281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:35.145257    5831 generic.go:356] "Generic (PLEG): container finished" podID="9ce43062-cb66-4466-8a10-f7c022b16588" containerID="7ac77218c0eda44add5e7698fea453664782cccc77fa321fdd340e2fa2032509" exitCode=0
Oct 31 03:10:35.145373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:35.145327    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6" event={"ID":"9ce43062-cb66-4466-8a10-f7c022b16588","Type":"ContainerDied","Data":"7ac77218c0eda44add5e7698fea453664782cccc77fa321fdd340e2fa2032509"}
Oct 31 03:10:36.155099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:36.155062    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerID="b69fee2447e555ddd76c20acd1f514acd823c9b925a99166cae396b06aa71555" exitCode=0
Oct 31 03:10:36.155519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:36.155263    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh" event={"ID":"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a","Type":"ContainerDied","Data":"b69fee2447e555ddd76c20acd1f514acd823c9b925a99166cae396b06aa71555"}
Oct 31 03:10:36.157994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:36.157884    5831 generic.go:356] "Generic (PLEG): container finished" podID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerID="269aa82d99a7f897df1bf637d5e0a020bbaaaa82fa8dc34c183d79d1f255d8c4" exitCode=0
Oct 31 03:10:36.157994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:36.157940    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc" event={"ID":"5d2645bc-3f45-46bf-9998-bc27374c789f","Type":"ContainerDied","Data":"269aa82d99a7f897df1bf637d5e0a020bbaaaa82fa8dc34c183d79d1f255d8c4"}
Oct 31 03:10:37.168098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.168072    5831 generic.go:356] "Generic (PLEG): container finished" podID="8ae1928b-abe2-4033-868b-a126462851db" containerID="d94865aa330360f71c846efca3714f4aadc3455fcedd175c59dded3749eb39ae" exitCode=0
Oct 31 03:10:37.168405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.168191    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv" event={"ID":"8ae1928b-abe2-4033-868b-a126462851db","Type":"ContainerDied","Data":"d94865aa330360f71c846efca3714f4aadc3455fcedd175c59dded3749eb39ae"}
Oct 31 03:10:37.172046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.172019    5831 generic.go:356] "Generic (PLEG): container finished" podID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerID="13257b297a04c7c3b70c79f37f75f015a55cf584d5416bfdb08aba19b40abb67" exitCode=0
Oct 31 03:10:37.172201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.172095    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerDied","Data":"13257b297a04c7c3b70c79f37f75f015a55cf584d5416bfdb08aba19b40abb67"}
Oct 31 03:10:37.175449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.175429    5831 generic.go:356] "Generic (PLEG): container finished" podID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerID="889d64bb80440a9bde78acb940bf1acbc47df9d2189fd632680b05708f411a22" exitCode=0
Oct 31 03:10:37.175559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.175513    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2" event={"ID":"4cb0341d-6afa-46f2-8129-ec3c694a95a5","Type":"ContainerDied","Data":"889d64bb80440a9bde78acb940bf1acbc47df9d2189fd632680b05708f411a22"}
Oct 31 03:10:37.178107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.178049    5831 generic.go:356] "Generic (PLEG): container finished" podID="3848a046-c893-4d98-a779-80b6d84f52ed" containerID="ac2f33863f6c6f62065dedf737252e1d142d2453a7743ec384c1b4a2b1237e8a" exitCode=0
Oct 31 03:10:37.178158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:37.178118    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerDied","Data":"ac2f33863f6c6f62065dedf737252e1d142d2453a7743ec384c1b4a2b1237e8a"}
Oct 31 03:10:38.190890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:38.190850    5831 generic.go:356] "Generic (PLEG): container finished" podID="9ce43062-cb66-4466-8a10-f7c022b16588" containerID="9dcd979e34b2f47d28d3b57c889178883ea4d284ea9a5b30f17c2d4806513159" exitCode=0
Oct 31 03:10:38.191414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:38.191022    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6" event={"ID":"9ce43062-cb66-4466-8a10-f7c022b16588","Type":"ContainerDied","Data":"9dcd979e34b2f47d28d3b57c889178883ea4d284ea9a5b30f17c2d4806513159"}
Oct 31 03:10:43.009477 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:10:43.009306    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:10:52.346716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:52.346101    5831 generic.go:356] "Generic (PLEG): container finished" podID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerID="fb7f22b418be20ea67fb827c707af45c7720336e0b4fd0afa5c201a385f505cf" exitCode=0
Oct 31 03:10:52.346716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:52.346211    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerDied","Data":"fb7f22b418be20ea67fb827c707af45c7720336e0b4fd0afa5c201a385f505cf"}
Oct 31 03:10:52.351146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:52.351102    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:10:52.352543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:52.352200    5831 generic.go:356] "Generic (PLEG): container finished" podID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerID="9e71b08a95f175335c04f4ca51e01449d29f2b449535daca206d0637a9159f35" exitCode=1
Oct 31 03:10:52.352543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:52.352326    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerDied","Data":"9e71b08a95f175335c04f4ca51e01449d29f2b449535daca206d0637a9159f35"}
Oct 31 03:10:53.617808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.617675    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:10:53.652721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.652130    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:53.665665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.665627    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:53.738787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738749    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\" (UniqueName: \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\") pod \"064ad017-1388-4a9d-8a0e-00e7fae27aef\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") "
Oct 31 03:10:53.738974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738794    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\" (UniqueName: \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\") pod \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") "
Oct 31 03:10:53.738974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738837    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\") pod \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") "
Oct 31 03:10:53.738974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738865    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\") pod \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\" (UID: \"a3139e21-cc0c-4910-8a21-e99bc0e5662e\") "
Oct 31 03:10:53.738974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738932    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\") pod \"064ad017-1388-4a9d-8a0e-00e7fae27aef\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") "
Oct 31 03:10:53.739196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.738976    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\") pod \"064ad017-1388-4a9d-8a0e-00e7fae27aef\" (UID: \"064ad017-1388-4a9d-8a0e-00e7fae27aef\") "
Oct 31 03:10:53.741513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.741452    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle" pod "a3139e21-cc0c-4910-8a21-e99bc0e5662e" (UID: "a3139e21-cc0c-4910-8a21-e99bc0e5662e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:53.742921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.742875    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle" pod "064ad017-1388-4a9d-8a0e-00e7fae27aef" (UID: "064ad017-1388-4a9d-8a0e-00e7fae27aef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:53.745103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.745073    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql" pod "a3139e21-cc0c-4910-8a21-e99bc0e5662e" (UID: "a3139e21-cc0c-4910-8a21-e99bc0e5662e"). InnerVolumeSpecName "kube-api-access-qqmql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:53.747418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.747379    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util" pod "a3139e21-cc0c-4910-8a21-e99bc0e5662e" (UID: "a3139e21-cc0c-4910-8a21-e99bc0e5662e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:53.747908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.747873    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util" pod "064ad017-1388-4a9d-8a0e-00e7fae27aef" (UID: "064ad017-1388-4a9d-8a0e-00e7fae27aef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:53.752577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.752514    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj" pod "064ad017-1388-4a9d-8a0e-00e7fae27aef" (UID: "064ad017-1388-4a9d-8a0e-00e7fae27aef"). InnerVolumeSpecName "kube-api-access-bz5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:53.840049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.839975    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5zj\" (UniqueName: \"kubernetes.io/projected/064ad017-1388-4a9d-8a0e-00e7fae27aef-kube-api-access-bz5zj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:53.840049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.840015    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmql\" (UniqueName: \"kubernetes.io/projected/a3139e21-cc0c-4910-8a21-e99bc0e5662e-kube-api-access-qqmql\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:53.840049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.840060    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:53.840323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.840074    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3139e21-cc0c-4910-8a21-e99bc0e5662e-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:53.840323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.840086    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:53.840323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:53.840097    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/064ad017-1388-4a9d-8a0e-00e7fae27aef-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:54.370325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.370284    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df" event={"ID":"064ad017-1388-4a9d-8a0e-00e7fae27aef","Type":"ContainerDied","Data":"87c0dc4d152a8f8ddfbd77bca924a2d861bc3a89bac8afb272e1563bbaea37ae"}
Oct 31 03:10:54.370516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.370348    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c0dc4d152a8f8ddfbd77bca924a2d861bc3a89bac8afb272e1563bbaea37ae"
Oct 31 03:10:54.370516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.370438    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j95df"
Oct 31 03:10:54.372675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.372649    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:10:54.374660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.374328    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh" event={"ID":"a3139e21-cc0c-4910-8a21-e99bc0e5662e","Type":"ContainerDied","Data":"2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598"}
Oct 31 03:10:54.374660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.374373    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2273ff50666afb1177925b2cc74e743ecfb96c14cb96beb133ef2fdb58ba7598"
Oct 31 03:10:54.374660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:54.374461    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh"
Oct 31 03:10:55.395510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:55.393640    5831 generic.go:356] "Generic (PLEG): container finished" podID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerID="5edd7919b26503ace1f692f0b4ab370c18d77805ad862b8bdf0c59408b1a94cc" exitCode=0
Oct 31 03:10:55.395510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:55.393780    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc" event={"ID":"5d2645bc-3f45-46bf-9998-bc27374c789f","Type":"ContainerDied","Data":"5edd7919b26503ace1f692f0b4ab370c18d77805ad862b8bdf0c59408b1a94cc"}
Oct 31 03:10:56.401003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.400916    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerID="498bbecf91d574e14e6cddf1d27d2f81fd7a053ee07ee68a52358d0bc0f3ebd1" exitCode=0
Oct 31 03:10:56.401436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.400985    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh" event={"ID":"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a","Type":"ContainerDied","Data":"498bbecf91d574e14e6cddf1d27d2f81fd7a053ee07ee68a52358d0bc0f3ebd1"}
Oct 31 03:10:56.403390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.403361    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerStarted","Data":"2737eece18c35f08ac6316e1653624125419b8e495a124a73dae306c4dcb9bce"}
Oct 31 03:10:56.587634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.587607    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:56.616934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.616877    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" podStartSLOduration=3.262023163 podStartE2EDuration="42.616856099s" totalImagesPullingTime="39.354832936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:56.615478767 +0000 UTC m=+206.630040476" watchObservedRunningTime="2026-10-31 03:10:56.616856099 +0000 UTC m=+206.631417808"
Oct 31 03:10:56.674867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.674799    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\") pod \"5d2645bc-3f45-46bf-9998-bc27374c789f\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") "
Oct 31 03:10:56.674867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.674863    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\") pod \"5d2645bc-3f45-46bf-9998-bc27374c789f\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") "
Oct 31 03:10:56.675069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.674921    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\" (UniqueName: \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\") pod \"5d2645bc-3f45-46bf-9998-bc27374c789f\" (UID: \"5d2645bc-3f45-46bf-9998-bc27374c789f\") "
Oct 31 03:10:56.675231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.675207    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle" pod "5d2645bc-3f45-46bf-9998-bc27374c789f" (UID: "5d2645bc-3f45-46bf-9998-bc27374c789f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:56.677201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.677179    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt" pod "5d2645bc-3f45-46bf-9998-bc27374c789f" (UID: "5d2645bc-3f45-46bf-9998-bc27374c789f"). InnerVolumeSpecName "kube-api-access-zbclt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:56.680198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.680175    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util" pod "5d2645bc-3f45-46bf-9998-bc27374c789f" (UID: "5d2645bc-3f45-46bf-9998-bc27374c789f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:56.776136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.776110    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:56.776136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.776136    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbclt\" (UniqueName: \"kubernetes.io/projected/5d2645bc-3f45-46bf-9998-bc27374c789f-kube-api-access-zbclt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:56.776136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:56.776146    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d2645bc-3f45-46bf-9998-bc27374c789f-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:57.410423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.410377    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc" event={"ID":"5d2645bc-3f45-46bf-9998-bc27374c789f","Type":"ContainerDied","Data":"fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3"}
Oct 31 03:10:57.410933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.410388    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fnmgc"
Oct 31 03:10:57.410933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.410440    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3b5c12d3083a5855156f13cbe330b1c52684082091991f61b589739d532fb3"
Oct 31 03:10:57.412716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.412688    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerStarted","Data":"aea551a47bc8419b4d1db1c34d6380bd9a8ac4884f1906bbaf1924ec73457265"}
Oct 31 03:10:57.414745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.414721    5831 generic.go:356] "Generic (PLEG): container finished" podID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerID="cebfd9e8e6982268cbbf20c6ef3ccc5ab066c43dc3145c946717a76440fc68d2" exitCode=0
Oct 31 03:10:57.414834 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.414799    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2" event={"ID":"4cb0341d-6afa-46f2-8129-ec3c694a95a5","Type":"ContainerDied","Data":"cebfd9e8e6982268cbbf20c6ef3ccc5ab066c43dc3145c946717a76440fc68d2"}
Oct 31 03:10:57.416788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.416766    5831 generic.go:356] "Generic (PLEG): container finished" podID="9ce43062-cb66-4466-8a10-f7c022b16588" containerID="7ea9cba896ead26b25922d53c2e672d25fad61c80917790a6cf6637efd5cafb8" exitCode=0
Oct 31 03:10:57.416896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.416793    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6" event={"ID":"9ce43062-cb66-4466-8a10-f7c022b16588","Type":"ContainerDied","Data":"7ea9cba896ead26b25922d53c2e672d25fad61c80917790a6cf6637efd5cafb8"}
Oct 31 03:10:57.419095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.419075    5831 generic.go:356] "Generic (PLEG): container finished" podID="3848a046-c893-4d98-a779-80b6d84f52ed" containerID="2737eece18c35f08ac6316e1653624125419b8e495a124a73dae306c4dcb9bce" exitCode=0
Oct 31 03:10:57.419189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.419150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerDied","Data":"2737eece18c35f08ac6316e1653624125419b8e495a124a73dae306c4dcb9bce"}
Oct 31 03:10:57.421296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.421275    5831 generic.go:356] "Generic (PLEG): container finished" podID="8ae1928b-abe2-4033-868b-a126462851db" containerID="8e9bd1aee570d608c2554fa44c6dab608d6822a614bb8bb7a4dc080c51f0d57c" exitCode=0
Oct 31 03:10:57.421379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.421338    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv" event={"ID":"8ae1928b-abe2-4033-868b-a126462851db","Type":"ContainerDied","Data":"8e9bd1aee570d608c2554fa44c6dab608d6822a614bb8bb7a4dc080c51f0d57c"}
Oct 31 03:10:57.521295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.521247    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" podStartSLOduration=3.300123262 podStartE2EDuration="44.521219354s" totalImagesPullingTime="41.221096092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:57.520342753 +0000 UTC m=+207.534904487" watchObservedRunningTime="2026-10-31 03:10:57.521219354 +0000 UTC m=+207.535781062"
Oct 31 03:10:57.542760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.542734    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:57.682439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.682405    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\") pod \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") "
Oct 31 03:10:57.682676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.682569    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\") pod \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") "
Oct 31 03:10:57.682676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.682611    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\" (UniqueName: \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\") pod \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\" (UID: \"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a\") "
Oct 31 03:10:57.683309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.683282    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle" pod "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" (UID: "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:57.684851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.684832    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc" pod "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" (UID: "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a"). InnerVolumeSpecName "kube-api-access-hbvzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:57.688077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.688058    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util" pod "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" (UID: "4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:57.783645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.783513    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:57.783645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.783643    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvzc\" (UniqueName: \"kubernetes.io/projected/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-kube-api-access-hbvzc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:57.783803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:57.783659    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.428119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.428039    5831 generic.go:356] "Generic (PLEG): container finished" podID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerID="aea551a47bc8419b4d1db1c34d6380bd9a8ac4884f1906bbaf1924ec73457265" exitCode=0
Oct 31 03:10:58.428542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.428132    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerDied","Data":"aea551a47bc8419b4d1db1c34d6380bd9a8ac4884f1906bbaf1924ec73457265"}
Oct 31 03:10:58.429968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.429950    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh"
Oct 31 03:10:58.430077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.429967    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbb8qh" event={"ID":"4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a","Type":"ContainerDied","Data":"a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72"}
Oct 31 03:10:58.430077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.430005    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8acbf3c658d89fd1ee17262363f17795bdac5874f9b7fa76e73e61720eb7f72"
Oct 31 03:10:58.583084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.583057    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:58.621830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.621800    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:58.635174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.635149    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:58.638344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.638322    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:58.693014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.692926    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\") pod \"8ae1928b-abe2-4033-868b-a126462851db\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") "
Oct 31 03:10:58.693014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.692967    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\") pod \"9ce43062-cb66-4466-8a10-f7c022b16588\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") "
Oct 31 03:10:58.693014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.692998    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\" (UniqueName: \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\") pod \"9ce43062-cb66-4466-8a10-f7c022b16588\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693019    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\" (UniqueName: \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\") pod \"8ae1928b-abe2-4033-868b-a126462851db\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\") pod \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693068    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\") pod \"8ae1928b-abe2-4033-868b-a126462851db\" (UID: \"8ae1928b-abe2-4033-868b-a126462851db\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693097    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\") pod \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693148    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\" (UniqueName: \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\") pod \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\" (UID: \"4cb0341d-6afa-46f2-8129-ec3c694a95a5\") "
Oct 31 03:10:58.693305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693197    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\") pod \"9ce43062-cb66-4466-8a10-f7c022b16588\" (UID: \"9ce43062-cb66-4466-8a10-f7c022b16588\") "
Oct 31 03:10:58.693721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.693694    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle" pod "8ae1928b-abe2-4033-868b-a126462851db" (UID: "8ae1928b-abe2-4033-868b-a126462851db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.694485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.694191    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle" pod "4cb0341d-6afa-46f2-8129-ec3c694a95a5" (UID: "4cb0341d-6afa-46f2-8129-ec3c694a95a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.694617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.694568    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle" pod "9ce43062-cb66-4466-8a10-f7c022b16588" (UID: "9ce43062-cb66-4466-8a10-f7c022b16588"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.695808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.695779    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl" pod "9ce43062-cb66-4466-8a10-f7c022b16588" (UID: "9ce43062-cb66-4466-8a10-f7c022b16588"). InnerVolumeSpecName "kube-api-access-5nfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:58.695808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.695790    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn" pod "8ae1928b-abe2-4033-868b-a126462851db" (UID: "8ae1928b-abe2-4033-868b-a126462851db"). InnerVolumeSpecName "kube-api-access-tfxhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:58.695987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.695790    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf" pod "4cb0341d-6afa-46f2-8129-ec3c694a95a5" (UID: "4cb0341d-6afa-46f2-8129-ec3c694a95a5"). InnerVolumeSpecName "kube-api-access-cnkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:58.699227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.699200    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util" pod "4cb0341d-6afa-46f2-8129-ec3c694a95a5" (UID: "4cb0341d-6afa-46f2-8129-ec3c694a95a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.700963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.700934    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util" pod "8ae1928b-abe2-4033-868b-a126462851db" (UID: "8ae1928b-abe2-4033-868b-a126462851db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.701123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.701098    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util" pod "9ce43062-cb66-4466-8a10-f7c022b16588" (UID: "9ce43062-cb66-4466-8a10-f7c022b16588"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.794131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794096    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\") pod \"3848a046-c893-4d98-a779-80b6d84f52ed\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") "
Oct 31 03:10:58.794329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794172    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\") pod \"3848a046-c893-4d98-a779-80b6d84f52ed\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") "
Oct 31 03:10:58.794329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794195    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\" (UniqueName: \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\") pod \"3848a046-c893-4d98-a779-80b6d84f52ed\" (UID: \"3848a046-c893-4d98-a779-80b6d84f52ed\") "
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794333    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794345    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794353    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ce43062-cb66-4466-8a10-f7c022b16588-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794362    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfpl\" (UniqueName: \"kubernetes.io/projected/9ce43062-cb66-4466-8a10-f7c022b16588-kube-api-access-5nfpl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794372    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxhn\" (UniqueName: \"kubernetes.io/projected/8ae1928b-abe2-4033-868b-a126462851db-kube-api-access-tfxhn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794383    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794390    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae1928b-abe2-4033-868b-a126462851db-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794397    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cb0341d-6afa-46f2-8129-ec3c694a95a5-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.794458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.794408    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkqf\" (UniqueName: \"kubernetes.io/projected/4cb0341d-6afa-46f2-8129-ec3c694a95a5-kube-api-access-cnkqf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.795031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.795004    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle" pod "3848a046-c893-4d98-a779-80b6d84f52ed" (UID: "3848a046-c893-4d98-a779-80b6d84f52ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.796511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.796491    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr" pod "3848a046-c893-4d98-a779-80b6d84f52ed" (UID: "3848a046-c893-4d98-a779-80b6d84f52ed"). InnerVolumeSpecName "kube-api-access-g5dwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:58.799818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.799796    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util" pod "3848a046-c893-4d98-a779-80b6d84f52ed" (UID: "3848a046-c893-4d98-a779-80b6d84f52ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:58.895388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.895355    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.895388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.895383    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dwr\" (UniqueName: \"kubernetes.io/projected/3848a046-c893-4d98-a779-80b6d84f52ed-kube-api-access-g5dwr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:58.895388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:58.895392    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3848a046-c893-4d98-a779-80b6d84f52ed-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:59.136951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.136860    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"]
Oct 31 03:10:59.137250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137235    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="extract"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137252    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="extract"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137262    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="util"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137267    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="util"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137275    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="util"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137285    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="util"
Oct 31 03:10:59.137292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137292    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137296    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137306    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137311    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137317    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137321    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137327    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137331    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137337    5831 cpu_manager.go:414] "containerMap: removing container" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137341    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137346    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137350    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137360    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137364    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137369    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137374    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137381    5831 cpu_manager.go:414] "containerMap: removing container" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137385    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137399    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137403    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137409    5831 cpu_manager.go:414] "containerMap: removing container" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137413    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137418    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137422    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137429    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137433    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137439    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137442    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137450    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137454    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137459    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137462    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137476    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137481    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137489    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137493    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="util"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137499    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137503    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137508    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137512    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="extract"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137520    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="pull"
Oct 31 03:10:59.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137524    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="pull"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137620    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3139e21-cc0c-4910-8a21-e99bc0e5662e" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137628    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3848a046-c893-4d98-a779-80b6d84f52ed" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137635    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce43062-cb66-4466-8a10-f7c022b16588" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137640    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb0341d-6afa-46f2-8129-ec3c694a95a5" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137645    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae1928b-abe2-4033-868b-a126462851db" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137650    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="064ad017-1388-4a9d-8a0e-00e7fae27aef" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137656    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2645bc-3f45-46bf-9998-bc27374c789f" containerName="extract"
Oct 31 03:10:59.138673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.137663    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad51f1e-3f7c-4593-8bb1-2e3e5e18396a" containerName="extract"
Oct 31 03:10:59.142378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.142361    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.195520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195482    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 03:10:59.195520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195495    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 03:10:59.195520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195516    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 03:10:59.195834 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195626    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-92w4t\""
Oct 31 03:10:59.195834 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195675    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 03:10:59.195834 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.195746    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 03:10:59.244576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.244523    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"]
Oct 31 03:10:59.299383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299351    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.299561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299394    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.299561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299482    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.299643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299585    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.299643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299611    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.299706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.299647    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2cf\" (UniqueName: \"kubernetes.io/projected/bf78aad5-ed75-4777-a584-0f07c57e3841-kube-api-access-br2cf\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.400942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.400859    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.400942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.400895    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.400942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.400927    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2cf\" (UniqueName: \"kubernetes.io/projected/bf78aad5-ed75-4777-a584-0f07c57e3841-kube-api-access-br2cf\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.401232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.400952    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.401232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.400983    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.401232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.401003    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.401503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.401470    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.404100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.404074    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf78aad5-ed75-4777-a584-0f07c57e3841-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.404335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.404314    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.404408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.404340    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.404408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.404352    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf78aad5-ed75-4777-a584-0f07c57e3841-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.436193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.436158    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv"
Oct 31 03:10:59.436678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.436167    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gsmjv" event={"ID":"8ae1928b-abe2-4033-868b-a126462851db","Type":"ContainerDied","Data":"304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c"}
Oct 31 03:10:59.436678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.436294    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304b8649add4127aad81b1476b8b38076394c590b548b713750d1f79c87a0f8c"
Oct 31 03:10:59.438296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.438272    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2"
Oct 31 03:10:59.438433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.438277    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwtf2" event={"ID":"4cb0341d-6afa-46f2-8129-ec3c694a95a5","Type":"ContainerDied","Data":"feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb"}
Oct 31 03:10:59.438433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.438392    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feceef302383a5305a3e49e4db6ebd1be2236dd7356d9c7eca1468c0b947eacb"
Oct 31 03:10:59.440250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.440233    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6"
Oct 31 03:10:59.440360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.440266    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pctp6" event={"ID":"9ce43062-cb66-4466-8a10-f7c022b16588","Type":"ContainerDied","Data":"d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e"}
Oct 31 03:10:59.440360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.440308    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9a9cdb8d8dba8b7eecd89491dfcfdfd00a5b4e113a10f8e44444a9789ce947e"
Oct 31 03:10:59.442253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.442235    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9"
Oct 31 03:10:59.442353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.442246    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljpj9" event={"ID":"3848a046-c893-4d98-a779-80b6d84f52ed","Type":"ContainerDied","Data":"24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f"}
Oct 31 03:10:59.442353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.442287    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24870400c606db25d6476573e9b22a9c12e0180560b06b0d0beec9fbf386be2f"
Oct 31 03:10:59.454638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.454618    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2cf\" (UniqueName: \"kubernetes.io/projected/bf78aad5-ed75-4777-a584-0f07c57e3841-kube-api-access-br2cf\") pod \"opendatahub-operator-controller-manager-6f949c7964-lgm6x\" (UID: \"bf78aad5-ed75-4777-a584-0f07c57e3841\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.460441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.460414    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:10:59.569714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.569689    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:10:59.703628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.703595    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\" (UniqueName: \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\") pod \"f4588f57-e259-4eef-9188-2cf6c8321f22\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") "
Oct 31 03:10:59.703812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.703650    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\") pod \"f4588f57-e259-4eef-9188-2cf6c8321f22\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") "
Oct 31 03:10:59.703812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.703669    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\") pod \"f4588f57-e259-4eef-9188-2cf6c8321f22\" (UID: \"f4588f57-e259-4eef-9188-2cf6c8321f22\") "
Oct 31 03:10:59.704208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.704180    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle" pod "f4588f57-e259-4eef-9188-2cf6c8321f22" (UID: "f4588f57-e259-4eef-9188-2cf6c8321f22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:59.705836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.705812    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n" pod "f4588f57-e259-4eef-9188-2cf6c8321f22" (UID: "f4588f57-e259-4eef-9188-2cf6c8321f22"). InnerVolumeSpecName "kube-api-access-wmq8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:10:59.709162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.709140    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util" pod "f4588f57-e259-4eef-9188-2cf6c8321f22" (UID: "f4588f57-e259-4eef-9188-2cf6c8321f22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:10:59.805242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.805202    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmq8n\" (UniqueName: \"kubernetes.io/projected/f4588f57-e259-4eef-9188-2cf6c8321f22-kube-api-access-wmq8n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:59.805242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.805235    5831 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-util\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:59.805242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.805248    5831 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4588f57-e259-4eef-9188-2cf6c8321f22-bundle\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:10:59.949566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:10:59.949546    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"]
Oct 31 03:10:59.952431 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:10:59.952402    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78aad5_ed75_4777_a584_0f07c57e3841.slice/crio-3c362655944a2c161a8b0fcf2e58893bdcd40f9ab496a27a1a6e7782207297de WatchSource:0}: Error finding container 3c362655944a2c161a8b0fcf2e58893bdcd40f9ab496a27a1a6e7782207297de: Status 404 returned error can't find the container with id 3c362655944a2c161a8b0fcf2e58893bdcd40f9ab496a27a1a6e7782207297de
Oct 31 03:11:00.448983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:00.448959    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx"
Oct 31 03:11:00.449453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:00.448954    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chclvx" event={"ID":"f4588f57-e259-4eef-9188-2cf6c8321f22","Type":"ContainerDied","Data":"7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2"}
Oct 31 03:11:00.449453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:00.449094    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b532ff7eb8d114d01a982a7584049bdb0629d689f16dbb977e1d7599967a1d2"
Oct 31 03:11:00.450386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:00.450362    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x" event={"ID":"bf78aad5-ed75-4777-a584-0f07c57e3841","Type":"ContainerStarted","Data":"3c362655944a2c161a8b0fcf2e58893bdcd40f9ab496a27a1a6e7782207297de"}
Oct 31 03:11:09.307781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.307741    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"]
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308223    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="extract"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308241    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="extract"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308265    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="util"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308270    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="util"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308288    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="pull"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308295    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="pull"
Oct 31 03:11:09.310340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.308359    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4588f57-e259-4eef-9188-2cf6c8321f22" containerName="extract"
Oct 31 03:11:09.311325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.311306    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.360062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.360032    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"]
Oct 31 03:11:09.387616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.387586    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbgl\" (UniqueName: \"kubernetes.io/projected/f9600f80-b1d9-4f9c-9793-14e0607cda75-kube-api-access-blbgl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.387784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.387628    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9600f80-b1d9-4f9c-9793-14e0607cda75-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.404139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.404111    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:11:09.404139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.404122    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 03:11:09.404332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.404272    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7cxlq\""
Oct 31 03:11:09.488767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.488733    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbgl\" (UniqueName: \"kubernetes.io/projected/f9600f80-b1d9-4f9c-9793-14e0607cda75-kube-api-access-blbgl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.488966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.488781    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9600f80-b1d9-4f9c-9793-14e0607cda75-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.489299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.489279    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9600f80-b1d9-4f9c-9793-14e0607cda75-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.549162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.549127    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbgl\" (UniqueName: \"kubernetes.io/projected/f9600f80-b1d9-4f9c-9793-14e0607cda75-kube-api-access-blbgl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l\" (UID: \"f9600f80-b1d9-4f9c-9793-14e0607cda75\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:09.623660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:09.623568    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"
Oct 31 03:11:10.100687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:10.100654    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l"]
Oct 31 03:11:10.499009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:10.498966    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x" event={"ID":"bf78aad5-ed75-4777-a584-0f07c57e3841","Type":"ContainerStarted","Data":"6d1027263e36524e122dcc2e54ed0d488faab65cc90e60a6b2518df690dde550"}
Oct 31 03:11:10.500414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:10.500387    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l" event={"ID":"f9600f80-b1d9-4f9c-9793-14e0607cda75","Type":"ContainerStarted","Data":"bc31f4d39a874c6a1905a3af5a73b724923ea4dbb07865461a527cd0e3656c6f"}
Oct 31 03:11:11.506458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:11.506422    5831 generic.go:356] "Generic (PLEG): container finished" podID="bf78aad5-ed75-4777-a584-0f07c57e3841" containerID="6d1027263e36524e122dcc2e54ed0d488faab65cc90e60a6b2518df690dde550" exitCode=0
Oct 31 03:11:11.506892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:11.506509    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x" event={"ID":"bf78aad5-ed75-4777-a584-0f07c57e3841","Type":"ContainerDied","Data":"6d1027263e36524e122dcc2e54ed0d488faab65cc90e60a6b2518df690dde550"}
Oct 31 03:11:12.515033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:12.515003    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x" event={"ID":"bf78aad5-ed75-4777-a584-0f07c57e3841","Type":"ContainerStarted","Data":"1c7afc676fee3d53589c2800323adef88a154a07789d10d37c772274b567dade"}
Oct 31 03:11:12.515463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:12.515184    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:11:12.738931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:12.738859    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x" podStartSLOduration=2.053254509 podStartE2EDuration="13.738840333s" totalImagesPullingTime="11.261697299s" totalInitContainerRuntime="423.888525ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:12.692041174 +0000 UTC m=+222.706602884" watchObservedRunningTime="2026-10-31 03:11:12.738840333 +0000 UTC m=+222.753402043"
Oct 31 03:11:12.976919 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:11:12.976876    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:11:13.529449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:13.529412    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l" event={"ID":"f9600f80-b1d9-4f9c-9793-14e0607cda75","Type":"ContainerStarted","Data":"e10bbc0f0f94d866b745b78637e0043b8c078e23a15cf62b28fe43ed55b66f62"}
Oct 31 03:11:13.641363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:13.641293    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mwh7l" podStartSLOduration=1.96440205 podStartE2EDuration="4.641279925s" totalImagesPullingTime="2.676877875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:13.640576 +0000 UTC m=+223.655137701" watchObservedRunningTime="2026-10-31 03:11:13.641279925 +0000 UTC m=+223.655841633"
Oct 31 03:11:22.380671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.380627    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"]
Oct 31 03:11:22.386157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.386132    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.404551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.404499    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.404718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.404693    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tfg\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-kube-api-access-g9tfg\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.452186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.452142    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"]
Oct 31 03:11:22.495831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.495797    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 03:11:22.495989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.495905    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 03:11:22.495989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.495972    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lbtpm\""
Oct 31 03:11:22.505583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.505517    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tfg\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-kube-api-access-g9tfg\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.505746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.505596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.559619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.559591    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tfg\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-kube-api-access-g9tfg\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.559765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.559644    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rhsg5\" (UID: \"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:22.713643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:22.713604    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"
Oct 31 03:11:23.078191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:23.078076    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5"]
Oct 31 03:11:23.081189 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:23.081160    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e79ccb7_460d_4d6e_921e_e9f9f0cf47ff.slice/crio-c3ac193d4a117cd3f3de84f25b30c20035872cfcd7e6b1c1a04e1dde1c8ac620 WatchSource:0}: Error finding container c3ac193d4a117cd3f3de84f25b30c20035872cfcd7e6b1c1a04e1dde1c8ac620: Status 404 returned error can't find the container with id c3ac193d4a117cd3f3de84f25b30c20035872cfcd7e6b1c1a04e1dde1c8ac620
Oct 31 03:11:23.531690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:23.531660    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-lgm6x"
Oct 31 03:11:23.578940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:23.578906    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5" event={"ID":"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff","Type":"ContainerStarted","Data":"c3ac193d4a117cd3f3de84f25b30c20035872cfcd7e6b1c1a04e1dde1c8ac620"}
Oct 31 03:11:24.211358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.211318    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:11:24.216128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.216109    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.266130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.266095    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c6vvr\""
Oct 31 03:11:24.266769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.266739    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 03:11:24.266901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.266772    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 03:11:24.272442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.272419    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:11:24.321361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.321326    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.321542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.321373    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hx4\" (UniqueName: \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.422032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.422002    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.422213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.422041    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hx4\" (UniqueName: \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.422453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.422433    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.475536 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.475447    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hx4\" (UniqueName: \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hnxxr\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.529618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.529580    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:24.997946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:24.997910    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:11:25.001853 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:25.001815    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0839d9_a7f3_4de3_9d11_109ec9dd7688.slice/crio-8e91a35900702ea5253e05d51157fdc1022a5415a6e4e944e2fba91e9fa4c6ac WatchSource:0}: Error finding container 8e91a35900702ea5253e05d51157fdc1022a5415a6e4e944e2fba91e9fa4c6ac: Status 404 returned error can't find the container with id 8e91a35900702ea5253e05d51157fdc1022a5415a6e4e944e2fba91e9fa4c6ac
Oct 31 03:11:25.593541 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:25.593499    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" event={"ID":"cc0839d9-a7f3-4de3-9d11-109ec9dd7688","Type":"ContainerStarted","Data":"8e91a35900702ea5253e05d51157fdc1022a5415a6e4e944e2fba91e9fa4c6ac"}
Oct 31 03:11:27.608146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.608106    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5" event={"ID":"3e79ccb7-460d-4d6e-921e-e9f9f0cf47ff","Type":"ContainerStarted","Data":"5e670dd36e3827177b1c6d2358cbccdf47fc4a388fe78af4a767579c30e6cece"}
Oct 31 03:11:27.714366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.712880    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"]
Oct 31 03:11:27.717224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.717199    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:27.759488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.759433    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fz8l\" (UniqueName: \"kubernetes.io/projected/7da6fd01-0c72-4691-b956-6807e32461c1-kube-api-access-8fz8l\") pod \"dns-operator-controller-manager-8557cbdb84-mxvkt\" (UID: \"7da6fd01-0c72-4691-b956-6807e32461c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:27.769500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.769471    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"]
Oct 31 03:11:27.771307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.771281    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 03:11:27.771468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.771336    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pnw7b\""
Oct 31 03:11:27.813670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.813596    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rhsg5" podStartSLOduration=2.107965332 podStartE2EDuration="5.813574724s" totalImagesPullingTime="3.705609392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:27.812745423 +0000 UTC m=+237.827307134" watchObservedRunningTime="2026-10-31 03:11:27.813574724 +0000 UTC m=+237.828136433"
Oct 31 03:11:27.860761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.860661    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fz8l\" (UniqueName: \"kubernetes.io/projected/7da6fd01-0c72-4691-b956-6807e32461c1-kube-api-access-8fz8l\") pod \"dns-operator-controller-manager-8557cbdb84-mxvkt\" (UID: \"7da6fd01-0c72-4691-b956-6807e32461c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:27.924656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:27.924614    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fz8l\" (UniqueName: \"kubernetes.io/projected/7da6fd01-0c72-4691-b956-6807e32461c1-kube-api-access-8fz8l\") pod \"dns-operator-controller-manager-8557cbdb84-mxvkt\" (UID: \"7da6fd01-0c72-4691-b956-6807e32461c1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:28.035982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:28.035945    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:28.426106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:28.425932    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"]
Oct 31 03:11:28.430714 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:28.429620    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da6fd01_0c72_4691_b956_6807e32461c1.slice/crio-b524323d3eace65e16d4635dd79dc671a69f6760b8ff9efac74d0287035a0ef0 WatchSource:0}: Error finding container b524323d3eace65e16d4635dd79dc671a69f6760b8ff9efac74d0287035a0ef0: Status 404 returned error can't find the container with id b524323d3eace65e16d4635dd79dc671a69f6760b8ff9efac74d0287035a0ef0
Oct 31 03:11:28.620130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:28.620023    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt" event={"ID":"7da6fd01-0c72-4691-b956-6807e32461c1","Type":"ContainerStarted","Data":"b524323d3eace65e16d4635dd79dc671a69f6760b8ff9efac74d0287035a0ef0"}
Oct 31 03:11:31.463727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.463693    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 03:11:31.637862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.637745    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" event={"ID":"cc0839d9-a7f3-4de3-9d11-109ec9dd7688","Type":"ContainerStarted","Data":"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"}
Oct 31 03:11:31.637862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.637839    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:31.639760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.639734    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt" event={"ID":"7da6fd01-0c72-4691-b956-6807e32461c1","Type":"ContainerStarted","Data":"e2acb8e91b57fe2f10c2763812ac384ed9167b106447d9aca7eab0bf43fcc94b"}
Oct 31 03:11:31.639895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.639787    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:31.962971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.962920    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" podStartSLOduration=1.667760646 podStartE2EDuration="7.962905027s" totalImagesPullingTime="6.295144381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:31.852869912 +0000 UTC m=+241.867431619" watchObservedRunningTime="2026-10-31 03:11:31.962905027 +0000 UTC m=+241.977466738"
Oct 31 03:11:31.963305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:31.963249    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt" podStartSLOduration=1.9806070359999999 podStartE2EDuration="4.963239868s" totalImagesPullingTime="2.982632832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:31.961855494 +0000 UTC m=+241.976417201" watchObservedRunningTime="2026-10-31 03:11:31.963239868 +0000 UTC m=+241.977801577"
Oct 31 03:11:33.809494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:33.809462    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"]
Oct 31 03:11:33.813065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:33.813044    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:33.862498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:33.862466    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z5bmc\""
Oct 31 03:11:33.909841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:33.909811    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"]
Oct 31 03:11:33.912158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:33.912136    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmx4\" (UniqueName: \"kubernetes.io/projected/b13d7e68-3328-4ef1-9449-fddfcce22cea-kube-api-access-gzmx4\") pod \"authorino-operator-7b7bb4bc77-dwqhg\" (UID: \"b13d7e68-3328-4ef1-9449-fddfcce22cea\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:34.012879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:34.012838    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmx4\" (UniqueName: \"kubernetes.io/projected/b13d7e68-3328-4ef1-9449-fddfcce22cea-kube-api-access-gzmx4\") pod \"authorino-operator-7b7bb4bc77-dwqhg\" (UID: \"b13d7e68-3328-4ef1-9449-fddfcce22cea\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:34.067603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:34.067514    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmx4\" (UniqueName: \"kubernetes.io/projected/b13d7e68-3328-4ef1-9449-fddfcce22cea-kube-api-access-gzmx4\") pod \"authorino-operator-7b7bb4bc77-dwqhg\" (UID: \"b13d7e68-3328-4ef1-9449-fddfcce22cea\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:34.125130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:34.125088    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:34.510373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:34.510349    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"]
Oct 31 03:11:34.513078 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:34.513045    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13d7e68_3328_4ef1_9449_fddfcce22cea.slice/crio-d53a5613d029079c6a8131c013b1ffc8cb0541f966940b3aea1a9a305228b01b WatchSource:0}: Error finding container d53a5613d029079c6a8131c013b1ffc8cb0541f966940b3aea1a9a305228b01b: Status 404 returned error can't find the container with id d53a5613d029079c6a8131c013b1ffc8cb0541f966940b3aea1a9a305228b01b
Oct 31 03:11:34.661041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:34.661007    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg" event={"ID":"b13d7e68-3328-4ef1-9449-fddfcce22cea","Type":"ContainerStarted","Data":"d53a5613d029079c6a8131c013b1ffc8cb0541f966940b3aea1a9a305228b01b"}
Oct 31 03:11:37.679922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:37.679889    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg" event={"ID":"b13d7e68-3328-4ef1-9449-fddfcce22cea","Type":"ContainerStarted","Data":"58fc9862bc8452cae03405de730fc1f844cada2347cd3a659ed2d97555f4ef68"}
Oct 31 03:11:37.680318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:37.680030    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:37.787905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:37.787854    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg" podStartSLOduration=2.06037561 podStartE2EDuration="4.787840893s" totalImagesPullingTime="2.727465283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:37.787364886 +0000 UTC m=+247.801926595" watchObservedRunningTime="2026-10-31 03:11:37.787840893 +0000 UTC m=+247.802402601"
Oct 31 03:11:42.647038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:42.647005    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mxvkt"
Oct 31 03:11:42.647512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:42.647094    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:42.977370 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:11:42.977330    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:11:43.992780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:43.992718    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 03:11:43.997661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:43.997639    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.095426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.095392    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 03:11:44.103946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.103915    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.104130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.103984    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cj4\" (UniqueName: \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.205604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.205561    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.205847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.205654    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cj4\" (UniqueName: \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.206017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.205999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.266443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.266375    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cj4\" (UniqueName: \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\") pod \"kuadrant-operator-controller-manager-678d87766f-ss2vc\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.309799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.309759    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:44.594318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.594236    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"]
Oct 31 03:11:44.598121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.598098    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.662133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.662104    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-c5j4f\""
Oct 31 03:11:44.663875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.663850    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"]
Oct 31 03:11:44.710132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710098    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710156    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710272    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710340    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710384    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710413    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710454    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710519    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.710619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.710584    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lsh\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-kube-api-access-l4lsh\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.744188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.744157    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 03:11:44.747461 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:44.747431    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3778b1c5_f1c0_43d6_b45c_c121a45904d2.slice/crio-db4e2627b788600e71f8b6eac69a81bae83dc76e1c13c9038f56dbc25a351029 WatchSource:0}: Error finding container db4e2627b788600e71f8b6eac69a81bae83dc76e1c13c9038f56dbc25a351029: Status 404 returned error can't find the container with id db4e2627b788600e71f8b6eac69a81bae83dc76e1c13c9038f56dbc25a351029
Oct 31 03:11:44.811800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.811766    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.811800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.811802    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.811824    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.811955    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812011    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812053    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lsh\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-kube-api-access-l4lsh\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812088    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812171    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812195    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812203    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812651    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812705    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.812754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.812724    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.814763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.814737    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.814870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.814780    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.865933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.865865    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.865933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.865920    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lsh\" (UniqueName: \"kubernetes.io/projected/8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e-kube-api-access-l4lsh\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5slg5v\" (UID: \"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:44.910565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:44.910517    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:45.296408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.296371    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"]
Oct 31 03:11:45.301085 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:11:45.301053    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8bf850_49a7_4b6c_ae2f_c7daafc7c61e.slice/crio-ee39da2c649d7bf211d7d16c6c227245aed38472e869cb0ed4f1e28ce5555389 WatchSource:0}: Error finding container ee39da2c649d7bf211d7d16c6c227245aed38472e869cb0ed4f1e28ce5555389: Status 404 returned error can't find the container with id ee39da2c649d7bf211d7d16c6c227245aed38472e869cb0ed4f1e28ce5555389
Oct 31 03:11:45.304065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.304027    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:11:45.304189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.304134    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:11:45.304247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.304198    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:11:45.724077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.724033    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v" event={"ID":"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e","Type":"ContainerStarted","Data":"479eaa98394bd9cc2a9dafbb1d14303350a9d3a2e8bb4f9870ed3dde058178cc"}
Oct 31 03:11:45.724265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.724100    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v" event={"ID":"8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e","Type":"ContainerStarted","Data":"ee39da2c649d7bf211d7d16c6c227245aed38472e869cb0ed4f1e28ce5555389"}
Oct 31 03:11:45.726038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.726005    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" event={"ID":"3778b1c5-f1c0-43d6-b45c-c121a45904d2","Type":"ContainerStarted","Data":"ecb0e550d411f360efc20a0d4c26756b35b6e8125224557a7dba7f092b769b0b"}
Oct 31 03:11:45.726184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.726059    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" event={"ID":"3778b1c5-f1c0-43d6-b45c-c121a45904d2","Type":"ContainerStarted","Data":"db4e2627b788600e71f8b6eac69a81bae83dc76e1c13c9038f56dbc25a351029"}
Oct 31 03:11:45.726512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.726496    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:45.839422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.839366    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v" podStartSLOduration=1.8393516829999998 podStartE2EDuration="1.839351683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:45.838892616 +0000 UTC m=+255.853454324" watchObservedRunningTime="2026-10-31 03:11:45.839351683 +0000 UTC m=+255.853913391"
Oct 31 03:11:45.911160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.911112    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:45.950681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:45.950616    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" podStartSLOduration=2.950600777 podStartE2EDuration="2.950600777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:45.948518204 +0000 UTC m=+255.963079934" watchObservedRunningTime="2026-10-31 03:11:45.950600777 +0000 UTC m=+255.965162484"
Oct 31 03:11:46.916111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:46.916082    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:47.735912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:47.735881    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:47.736797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:47.736776    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5slg5v"
Oct 31 03:11:48.687394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:48.687361    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dwqhg"
Oct 31 03:11:57.738041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:57.738006    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 03:11:57.949312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:57.949266    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:11:57.949597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:57.949549    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" podUID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" containerName="manager" containerID="cri-o://50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e" gracePeriod=10
Oct 31 03:11:58.202739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.202687    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:58.326810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.326720    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\") pod \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") "
Oct 31 03:11:58.326810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.326782    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\" (UniqueName: \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\") pod \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\" (UID: \"cc0839d9-a7f3-4de3-9d11-109ec9dd7688\") "
Oct 31 03:11:58.327344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.327313    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume" pod "cc0839d9-a7f3-4de3-9d11-109ec9dd7688" (UID: "cc0839d9-a7f3-4de3-9d11-109ec9dd7688"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:11:58.329599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.329572    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4" pod "cc0839d9-a7f3-4de3-9d11-109ec9dd7688" (UID: "cc0839d9-a7f3-4de3-9d11-109ec9dd7688"). InnerVolumeSpecName "kube-api-access-m9hx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:11:58.428320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.428278    5831 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-extensions-socket-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:11:58.428320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.428316    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9hx4\" (UniqueName: \"kubernetes.io/projected/cc0839d9-a7f3-4de3-9d11-109ec9dd7688-kube-api-access-m9hx4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:11:58.792860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.792823    5831 generic.go:356] "Generic (PLEG): container finished" podID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" containerID="50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e" exitCode=0
Oct 31 03:11:58.793369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.792874    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"
Oct 31 03:11:58.793369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.792919    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" event={"ID":"cc0839d9-a7f3-4de3-9d11-109ec9dd7688","Type":"ContainerDied","Data":"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"}
Oct 31 03:11:58.793369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.792997    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr" event={"ID":"cc0839d9-a7f3-4de3-9d11-109ec9dd7688","Type":"ContainerDied","Data":"8e91a35900702ea5253e05d51157fdc1022a5415a6e4e944e2fba91e9fa4c6ac"}
Oct 31 03:11:58.793369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.793034    5831 scope.go:122] "RemoveContainer" containerID="50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"
Oct 31 03:11:58.802558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.802518    5831 scope.go:122] "RemoveContainer" containerID="50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"
Oct 31 03:11:58.802857 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:11:58.802840    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e\": container with ID starting with 50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e not found: ID does not exist" containerID="50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"
Oct 31 03:11:58.802917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.802866    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e"} err="failed to get container status \"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e\": rpc error: code = NotFound desc = could not find container \"50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e\": container with ID starting with 50a5c930e562745648e7c82d4faa6d8209f29d70a46c2bf4105cd33a8069053e not found: ID does not exist"
Oct 31 03:11:58.967626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:58.967594    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:11:59.013894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:11:59.013861    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hnxxr"]
Oct 31 03:12:00.133344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:00.133313    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" path="/var/lib/kubelet/pods/cc0839d9-a7f3-4de3-9d11-109ec9dd7688/volumes"
Oct 31 03:12:09.342249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.342211    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"]
Oct 31 03:12:09.343695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.343670    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" containerName="manager"
Oct 31 03:12:09.343852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.343839    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" containerName="manager"
Oct 31 03:12:09.344104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.344091    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0839d9-a7f3-4de3-9d11-109ec9dd7688" containerName="manager"
Oct 31 03:12:09.351700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.351676    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.400585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.400524    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 03:12:09.400775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.400627    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 03:12:09.420174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.420139    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.420328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.420180    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q2v\" (UniqueName: \"kubernetes.io/projected/d7d376d2-9269-4725-9b05-bbad81da4a84-kube-api-access-m8q2v\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.420328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.420209    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7d376d2-9269-4725-9b05-bbad81da4a84-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.420328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.420304    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.520770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.520731    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7d376d2-9269-4725-9b05-bbad81da4a84-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.520770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.520770    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.521034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.520827    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.521034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.520847    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q2v\" (UniqueName: \"kubernetes.io/projected/d7d376d2-9269-4725-9b05-bbad81da4a84-kube-api-access-m8q2v\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.521259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.521233    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.521332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.521254    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7d376d2-9269-4725-9b05-bbad81da4a84-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.539228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.539207    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-k6fxc\""
Oct 31 03:12:09.539425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.539406    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 03:12:09.541325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.541308    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"]
Oct 31 03:12:09.542431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.542410    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7d376d2-9269-4725-9b05-bbad81da4a84-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.576192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.576162    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q2v\" (UniqueName: \"kubernetes.io/projected/d7d376d2-9269-4725-9b05-bbad81da4a84-kube-api-access-m8q2v\") pod \"kserve-module-controller-manager-7566cf5fc8-7pq8r\" (UID: \"d7d376d2-9269-4725-9b05-bbad81da4a84\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:09.665022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:09.664935    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:10.086829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:10.086777    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"]
Oct 31 03:12:10.089299 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:12:10.089275    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d376d2_9269_4725_9b05_bbad81da4a84.slice/crio-e79f324946529c26299ba1559932f5323a35a462fcc8acdd5e53dfbad4fa73b1 WatchSource:0}: Error finding container e79f324946529c26299ba1559932f5323a35a462fcc8acdd5e53dfbad4fa73b1: Status 404 returned error can't find the container with id e79f324946529c26299ba1559932f5323a35a462fcc8acdd5e53dfbad4fa73b1
Oct 31 03:12:10.863905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:10.863869    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r" event={"ID":"d7d376d2-9269-4725-9b05-bbad81da4a84","Type":"ContainerStarted","Data":"e79f324946529c26299ba1559932f5323a35a462fcc8acdd5e53dfbad4fa73b1"}
Oct 31 03:12:12.982770 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:12:12.981503    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:12:13.882634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:13.882595    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r" event={"ID":"d7d376d2-9269-4725-9b05-bbad81da4a84","Type":"ContainerStarted","Data":"a6366c0a982628f176f6f2fd0890fffa01f65d08ecfee482406316a979b4e7d8"}
Oct 31 03:12:13.882848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:13.882659    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:14.032039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:14.031986    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r" podStartSLOduration=1.815075857 podStartE2EDuration="5.031971646s" totalImagesPullingTime="3.216895789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:14.031023709 +0000 UTC m=+284.045585417" watchObservedRunningTime="2026-10-31 03:12:14.031971646 +0000 UTC m=+284.046533390"
Oct 31 03:12:18.554496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.554461    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:12:18.558167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.558147    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.594909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.594877    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-config-file\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.594909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.594912    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhlv\" (UniqueName: \"kubernetes.io/projected/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-kube-api-access-zfhlv\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.610923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.610896    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:12:18.655127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.655104    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 03:12:18.655287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.655143    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4298d\""
Oct 31 03:12:18.696358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.696328    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-config-file\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.696575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.696367    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhlv\" (UniqueName: \"kubernetes.io/projected/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-kube-api-access-zfhlv\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.696959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.696940    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-config-file\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.751274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.751244    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhlv\" (UniqueName: \"kubernetes.io/projected/2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc-kube-api-access-zfhlv\") pod \"limitador-limitador-7c9b87d574-7phrx\" (UID: \"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:18.836074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.835992    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:12:18.870386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:18.870357    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:19.267812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:19.267788    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:12:19.270426 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:12:19.270400    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cae7632_d1fb_4de5_ba4c_d1f12f0d55fc.slice/crio-fd7ffe8420de4e9a697868b9ddf785a1fdb96999f8abe8be377fe1172a09d60e WatchSource:0}: Error finding container fd7ffe8420de4e9a697868b9ddf785a1fdb96999f8abe8be377fe1172a09d60e: Status 404 returned error can't find the container with id fd7ffe8420de4e9a697868b9ddf785a1fdb96999f8abe8be377fe1172a09d60e
Oct 31 03:12:19.917310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:19.917276    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx" event={"ID":"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc","Type":"ContainerStarted","Data":"fd7ffe8420de4e9a697868b9ddf785a1fdb96999f8abe8be377fe1172a09d60e"}
Oct 31 03:12:20.133649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.133617    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:20.138289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.138258    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:20.186113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.186081    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:20.187088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.187069    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bjvt\""
Oct 31 03:12:20.209643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.209613    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6t9h\" (UniqueName: \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\") pod \"authorino-57f7f8659f-5xxbf\" (UID: \"89545f12-d5f4-43bb-85c5-b30b3ef92923\") " pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:20.310357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.310317    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6t9h\" (UniqueName: \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\") pod \"authorino-57f7f8659f-5xxbf\" (UID: \"89545f12-d5f4-43bb-85c5-b30b3ef92923\") " pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:20.365420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.365382    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6t9h\" (UniqueName: \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\") pod \"authorino-57f7f8659f-5xxbf\" (UID: \"89545f12-d5f4-43bb-85c5-b30b3ef92923\") " pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:20.450307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.450224    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:20.814464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.814297    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:20.817385 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:12:20.817311    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89545f12_d5f4_43bb_85c5_b30b3ef92923.slice/crio-8cc512b0ebbe08a037516a016261acad28b7cbf529222e3c6c6a9daeb71e855a WatchSource:0}: Error finding container 8cc512b0ebbe08a037516a016261acad28b7cbf529222e3c6c6a9daeb71e855a: Status 404 returned error can't find the container with id 8cc512b0ebbe08a037516a016261acad28b7cbf529222e3c6c6a9daeb71e855a
Oct 31 03:12:20.923588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:20.923554    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" event={"ID":"89545f12-d5f4-43bb-85c5-b30b3ef92923","Type":"ContainerStarted","Data":"8cc512b0ebbe08a037516a016261acad28b7cbf529222e3c6c6a9daeb71e855a"}
Oct 31 03:12:21.930894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:21.930857    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx" event={"ID":"2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc","Type":"ContainerStarted","Data":"addc855c02e04fa8c85beaa33716735e1c2c31faaa27726855563a71fc9786d3"}
Oct 31 03:12:21.931371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:21.931008    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:22.049738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:22.049677    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx" podStartSLOduration=2.011249563 podStartE2EDuration="4.049660956s" totalImagesPullingTime="2.038411393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:22.0491263 +0000 UTC m=+292.063688018" watchObservedRunningTime="2026-10-31 03:12:22.049660956 +0000 UTC m=+292.064222664"
Oct 31 03:12:24.951948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:24.951911    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" event={"ID":"89545f12-d5f4-43bb-85c5-b30b3ef92923","Type":"ContainerStarted","Data":"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"}
Oct 31 03:12:25.068253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:25.068197    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" podStartSLOduration=1.9164077000000002 podStartE2EDuration="5.068184098s" totalImagesPullingTime="3.151776398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:25.067459357 +0000 UTC m=+295.082021065" watchObservedRunningTime="2026-10-31 03:12:25.068184098 +0000 UTC m=+295.082745806"
Oct 31 03:12:26.047902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:26.047871    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:26.962498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:26.962457    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" podUID="89545f12-d5f4-43bb-85c5-b30b3ef92923" containerName="authorino" containerID="cri-o://d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d" gracePeriod=30
Oct 31 03:12:27.205801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.205771    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:27.274647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.274573    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\" (UniqueName: \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\") pod \"89545f12-d5f4-43bb-85c5-b30b3ef92923\" (UID: \"89545f12-d5f4-43bb-85c5-b30b3ef92923\") "
Oct 31 03:12:27.276757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.276727    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h" pod "89545f12-d5f4-43bb-85c5-b30b3ef92923" (UID: "89545f12-d5f4-43bb-85c5-b30b3ef92923"). InnerVolumeSpecName "kube-api-access-w6t9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:12:27.375926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.375889    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6t9h\" (UniqueName: \"kubernetes.io/projected/89545f12-d5f4-43bb-85c5-b30b3ef92923-kube-api-access-w6t9h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:12:27.968544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.968506    5831 generic.go:356] "Generic (PLEG): container finished" podID="89545f12-d5f4-43bb-85c5-b30b3ef92923" containerID="d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d" exitCode=0
Oct 31 03:12:27.968729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.968583    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5xxbf"
Oct 31 03:12:27.968729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.968602    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" event={"ID":"89545f12-d5f4-43bb-85c5-b30b3ef92923","Type":"ContainerDied","Data":"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"}
Oct 31 03:12:27.968729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.968649    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5xxbf" event={"ID":"89545f12-d5f4-43bb-85c5-b30b3ef92923","Type":"ContainerDied","Data":"8cc512b0ebbe08a037516a016261acad28b7cbf529222e3c6c6a9daeb71e855a"}
Oct 31 03:12:27.968729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.968672    5831 scope.go:122] "RemoveContainer" containerID="d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"
Oct 31 03:12:27.978809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.978789    5831 scope.go:122] "RemoveContainer" containerID="d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"
Oct 31 03:12:27.979085 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:12:27.979062    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d\": container with ID starting with d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d not found: ID does not exist" containerID="d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"
Oct 31 03:12:27.979129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:27.979097    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d"} err="failed to get container status \"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d\": rpc error: code = NotFound desc = could not find container \"d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d\": container with ID starting with d61724102f63c0dce39cc7c039cd97a5a2a69d41cd85b3fb911280033eda168d not found: ID does not exist"
Oct 31 03:12:28.120389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:28.120351    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:28.124130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:28.124103    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5xxbf"]
Oct 31 03:12:28.133187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:28.133163    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89545f12-d5f4-43bb-85c5-b30b3ef92923" path="/var/lib/kubelet/pods/89545f12-d5f4-43bb-85c5-b30b3ef92923/volumes"
Oct 31 03:12:30.159195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.159167    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:12:30.159589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.159248    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:12:30.169985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.169960    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:12:30.170396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.170374    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:12:30.173375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.173354    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:12:30.173938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:30.173919    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:12:32.939225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:32.939193    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7phrx"
Oct 31 03:12:34.888766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:12:34.888731    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7pq8r"
Oct 31 03:12:42.977961 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:12:42.977922    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:13:02.853979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.853937    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:02.854645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.854499    5831 cpu_manager.go:414] "containerMap: removing container" podUID="89545f12-d5f4-43bb-85c5-b30b3ef92923" containerName="authorino"
Oct 31 03:13:02.854645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.854519    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89545f12-d5f4-43bb-85c5-b30b3ef92923" containerName="authorino"
Oct 31 03:13:02.854779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.854661    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="89545f12-d5f4-43bb-85c5-b30b3ef92923" containerName="authorino"
Oct 31 03:13:02.857304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.857283    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.893816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.893786    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphm9\" (UniqueName: \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.893983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.893838    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.893983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.893952    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.952285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.952243    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 03:13:02.952285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.952243    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l2xw2\""
Oct 31 03:13:02.952285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.952244    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 03:13:02.952548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.952359    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 03:13:02.954375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.954351    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:02.994811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.994772    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphm9\" (UniqueName: \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.995018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.994831    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.995018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.994862    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.997652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.997613    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:02.997652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:02.997633    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:03.060341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.060313    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphm9\" (UniqueName: \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\") pod \"maas-controller-55c9cb6786-fld89\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") " pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:03.170521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.170428    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:03.529984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.529952    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 03:13:03.534851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.534829    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.540603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.540582    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:03.577723 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:13:03.577690    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17cfa01_371c_43e0_a9e1_16f2fb70a017.slice/crio-8e810f8f97f686f99847b78b3a1b1c45e5caf03159cda6fa957977f6ad2781fd WatchSource:0}: Error finding container 8e810f8f97f686f99847b78b3a1b1c45e5caf03159cda6fa957977f6ad2781fd: Status 404 returned error can't find the container with id 8e810f8f97f686f99847b78b3a1b1c45e5caf03159cda6fa957977f6ad2781fd
Oct 31 03:13:03.578958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.578941    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:13:03.591048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.591020    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 03:13:03.598597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.598572    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7wc\" (UniqueName: \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.598698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.598616    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.598783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.598766    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.699409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.699369    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7wc\" (UniqueName: \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.699639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.699420    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.699639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.699469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.702028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.702007    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.702114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.702039    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.753933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.753902    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7wc\" (UniqueName: \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\") pod \"maas-controller-7c946769f6-k59sv\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") " pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:03.849016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:03.848928    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:04.167032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:04.166949    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c9cb6786-fld89" event={"ID":"e17cfa01-371c-43e0-a9e1-16f2fb70a017","Type":"ContainerStarted","Data":"8e810f8f97f686f99847b78b3a1b1c45e5caf03159cda6fa957977f6ad2781fd"}
Oct 31 03:13:04.225462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:04.225431    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 03:13:04.228508 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:13:04.228476    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93c4102_0f0a_401f_90f4_dc7d22955787.slice/crio-c5100c16265405774cb10c204e1934e10247086e6e4eba73f1f8808f99eb1d4c WatchSource:0}: Error finding container c5100c16265405774cb10c204e1934e10247086e6e4eba73f1f8808f99eb1d4c: Status 404 returned error can't find the container with id c5100c16265405774cb10c204e1934e10247086e6e4eba73f1f8808f99eb1d4c
Oct 31 03:13:05.173260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:05.173226    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-k59sv" event={"ID":"f93c4102-0f0a-401f-90f4-dc7d22955787","Type":"ContainerStarted","Data":"c5100c16265405774cb10c204e1934e10247086e6e4eba73f1f8808f99eb1d4c"}
Oct 31 03:13:11.603296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.603254    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"]
Oct 31 03:13:11.607300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.607274    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.657605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.657521    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:13:11.657605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.657523    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:13:11.657813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.657616    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:13:11.657813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.657686    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r7bff\""
Oct 31 03:13:11.668037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668009    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668061    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9m2\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-kube-api-access-nb9m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668136    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668169    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668305    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.668354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.668333    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.712681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.712604    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"]
Oct 31 03:13:11.769491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769458    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769494    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769608    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9m2\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-kube-api-access-nb9m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769642    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769685    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.769784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.769711    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.770331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.770303    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.772291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.772266    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.772401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.772274    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.772509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.772493    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.772674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.772655    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.824370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.824328    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.824370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.824355    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9m2\" (UniqueName: \"kubernetes.io/projected/8d43e0c8-cc75-4e72-8308-8972e3a3e1f8-kube-api-access-nb9m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkrml\" (UID: \"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:11.919626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:11.919581    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:12.224200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.224115    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c9cb6786-fld89" event={"ID":"e17cfa01-371c-43e0-a9e1-16f2fb70a017","Type":"ContainerStarted","Data":"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"}
Oct 31 03:13:12.224200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.224181    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:12.225689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.225661    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-k59sv" event={"ID":"f93c4102-0f0a-401f-90f4-dc7d22955787","Type":"ContainerStarted","Data":"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"}
Oct 31 03:13:12.225828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.225719    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:12.338644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.338623    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"]
Oct 31 03:13:12.340949 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:13:12.340913    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d43e0c8_cc75_4e72_8308_8972e3a3e1f8.slice/crio-e6df019d656ae18b2126b02aee46b1e5d6d562f1df60408a5e31faf712e1f6c4 WatchSource:0}: Error finding container e6df019d656ae18b2126b02aee46b1e5d6d562f1df60408a5e31faf712e1f6c4: Status 404 returned error can't find the container with id e6df019d656ae18b2126b02aee46b1e5d6d562f1df60408a5e31faf712e1f6c4
Oct 31 03:13:12.384660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.384609    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55c9cb6786-fld89" podStartSLOduration=2.521509708 podStartE2EDuration="10.384590072s" totalImagesPullingTime="7.863080364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:12.382283664 +0000 UTC m=+342.396845364" watchObservedRunningTime="2026-10-31 03:13:12.384590072 +0000 UTC m=+342.399151781"
Oct 31 03:13:12.499813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:12.499709    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7c946769f6-k59sv" podStartSLOduration=2.278116368 podStartE2EDuration="9.499691839s" totalImagesPullingTime="7.221575471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:12.498264302 +0000 UTC m=+342.512826010" watchObservedRunningTime="2026-10-31 03:13:12.499691839 +0000 UTC m=+342.514253546"
Oct 31 03:13:12.979243 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:13:12.979204    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:13:13.232259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:13.232166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml" event={"ID":"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8","Type":"ContainerStarted","Data":"e6df019d656ae18b2126b02aee46b1e5d6d562f1df60408a5e31faf712e1f6c4"}
Oct 31 03:13:17.861893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:17.861863    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:13:17.862176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:17.861927    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:13:18.262071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:18.262034    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml" event={"ID":"8d43e0c8-cc75-4e72-8308-8972e3a3e1f8","Type":"ContainerStarted","Data":"96c9db21e2bf1a146699a0fad97a7265f9e4196914d4c646f58e8d4a0e863570"}
Oct 31 03:13:18.262266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:18.262245    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:18.264156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:18.264132    5831 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vkrml container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:13:18.264281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:18.264224    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml" podUID="8d43e0c8-cc75-4e72-8308-8972e3a3e1f8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:13:18.458051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:18.458001    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml" podStartSLOduration=1.939925996 podStartE2EDuration="7.457986638s" totalImagesPullingTime="5.518060642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:18.419375511 +0000 UTC m=+348.433937219" watchObservedRunningTime="2026-10-31 03:13:18.457986638 +0000 UTC m=+348.472548345"
Oct 31 03:13:19.267975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:19.267946    5831 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vkrml container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:13:19.268373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:19.268005    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml" podUID="8d43e0c8-cc75-4e72-8308-8972e3a3e1f8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:13:22.268954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:22.268925    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkrml"
Oct 31 03:13:23.234835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.234793    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 03:13:23.235010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.234896    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:23.489512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.489425    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:23.489993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.489653    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-55c9cb6786-fld89" podUID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" containerName="manager" containerID="cri-o://2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00" gracePeriod=30
Oct 31 03:13:23.762610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.762582    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:23.890522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.890491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\") pod \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") "
Oct 31 03:13:23.890705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.890572    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\") pod \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") "
Oct 31 03:13:23.890705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.890619    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\" (UniqueName: \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\") pod \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\" (UID: \"e17cfa01-371c-43e0-a9e1-16f2fb70a017\") "
Oct 31 03:13:23.893362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.893324    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert" pod "e17cfa01-371c-43e0-a9e1-16f2fb70a017" (UID: "e17cfa01-371c-43e0-a9e1-16f2fb70a017"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:13:23.893510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.893372    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs" pod "e17cfa01-371c-43e0-a9e1-16f2fb70a017" (UID: "e17cfa01-371c-43e0-a9e1-16f2fb70a017"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:13:23.893510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.893400    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9" pod "e17cfa01-371c-43e0-a9e1-16f2fb70a017" (UID: "e17cfa01-371c-43e0-a9e1-16f2fb70a017"). InnerVolumeSpecName "kube-api-access-qphm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:13:23.991425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.991392    5831 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-webhook-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:23.991425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.991418    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17cfa01-371c-43e0-a9e1-16f2fb70a017-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:23.991425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:23.991429    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphm9\" (UniqueName: \"kubernetes.io/projected/e17cfa01-371c-43e0-a9e1-16f2fb70a017-kube-api-access-qphm9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:24.297510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.297474    5831 generic.go:356] "Generic (PLEG): container finished" podID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" containerID="2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00" exitCode=0
Oct 31 03:13:24.297750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.297555    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c9cb6786-fld89"
Oct 31 03:13:24.297750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.297561    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c9cb6786-fld89" event={"ID":"e17cfa01-371c-43e0-a9e1-16f2fb70a017","Type":"ContainerDied","Data":"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"}
Oct 31 03:13:24.297750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.297627    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c9cb6786-fld89" event={"ID":"e17cfa01-371c-43e0-a9e1-16f2fb70a017","Type":"ContainerDied","Data":"8e810f8f97f686f99847b78b3a1b1c45e5caf03159cda6fa957977f6ad2781fd"}
Oct 31 03:13:24.297750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.297650    5831 scope.go:122] "RemoveContainer" containerID="2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"
Oct 31 03:13:24.306777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.306759    5831 scope.go:122] "RemoveContainer" containerID="2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"
Oct 31 03:13:24.307042 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:13:24.307027    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00\": container with ID starting with 2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00 not found: ID does not exist" containerID="2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"
Oct 31 03:13:24.307082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.307051    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00"} err="failed to get container status \"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00\": rpc error: code = NotFound desc = could not find container \"2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00\": container with ID starting with 2f0ea55c03e0cb1c4e34fc87e8b9ac7cac4212f60ac1ab87865d8996986cca00 not found: ID does not exist"
Oct 31 03:13:24.452551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.452491    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:24.457025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:24.457001    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55c9cb6786-fld89"]
Oct 31 03:13:26.134121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:26.134085    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" path="/var/lib/kubelet/pods/e17cfa01-371c-43e0-a9e1-16f2fb70a017/volumes"
Oct 31 03:13:42.979931 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:13:42.979888    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:13:49.590856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:49.590812    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:13:59.495764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.495732    5831 generic.go:356] "Generic (PLEG): container finished" podID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerID="3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c" exitCode=0
Oct 31 03:13:59.496225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.495806    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerDied","Data":"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"}
Oct 31 03:13:59.496225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.496170    5831 scope.go:122] "RemoveContainer" containerID="3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"
Oct 31 03:13:59.497223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.497195    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:13:59.497276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.497262    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:13:59.497316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:13:59.497299    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:14:00.510988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:00.510954    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerStarted","Data":"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"}
Oct 31 03:14:00.776663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:00.776570    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:14:00.777734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:00.777713    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-22gdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:14:00.777837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:00.777754    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 31 03:14:01.099775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:01.099635    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:14:01.776397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:01.776362    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-22gdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:14:01.776833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:01.776415    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 31 03:14:02.777304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:02.777262    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-22gdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:14:02.777736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:02.777322    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 31 03:14:03.777022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:03.776981    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-22gdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:14:03.777205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:03.777047    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 31 03:14:04.776677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:04.776643    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-22gdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:14:04.777077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:04.776698    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 31 03:14:05.780603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:05.780573    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:14:05.781580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:14:05.781557    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:14:12.977671 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:14:12.977630    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:14:42.977329 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:14:42.977287    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:15:00.161327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.161243    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:00.161803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.161786    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" containerName="manager"
Oct 31 03:15:00.161853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.161806    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" containerName="manager"
Oct 31 03:15:00.161889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.161883    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17cfa01-371c-43e0-a9e1-16f2fb70a017" containerName="manager"
Oct 31 03:15:00.164905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.164885    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.215728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.215685    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:15:00.215902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.215753    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:15:00.242897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.242863    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.243054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.242954    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9t4\" (UniqueName: \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.261663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.261635    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:15:00.263762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.263741    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:00.343972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.343939    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.344167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.343991    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9t4\" (UniqueName: \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.344675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.344656    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.398052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.398021    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9t4\" (UniqueName: \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\") pod \"maas-api-key-cleanup-29890275-lb9np\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.477668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.477639    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:00.850925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:00.850900    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:00.853207 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:15:00.853182    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0776cf_14b4_4a97_9cb9_b0c66e581fd5.slice/crio-3d30dfe4bd5394dd84afa7ca116174754c157499d7775184be9dcf8bef7a656a WatchSource:0}: Error finding container 3d30dfe4bd5394dd84afa7ca116174754c157499d7775184be9dcf8bef7a656a: Status 404 returned error can't find the container with id 3d30dfe4bd5394dd84afa7ca116174754c157499d7775184be9dcf8bef7a656a
Oct 31 03:15:01.837002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:01.836958    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerStarted","Data":"3d30dfe4bd5394dd84afa7ca116174754c157499d7775184be9dcf8bef7a656a"}
Oct 31 03:15:05.861940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:05.861904    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerStarted","Data":"5c941778071455f6d7d00e1917c4d8a3a04b9f2409a9c559a2ffb35c4abe07e0"}
Oct 31 03:15:05.968074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:05.968020    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" podStartSLOduration=1.336684288 podStartE2EDuration="5.968007284s" totalImagesPullingTime="4.631322996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:05.96696379 +0000 UTC m=+455.981525497" watchObservedRunningTime="2026-10-31 03:15:05.968007284 +0000 UTC m=+455.982568970"
Oct 31 03:15:06.868797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:06.868758    5831 generic.go:356] "Generic (PLEG): container finished" podID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerID="5c941778071455f6d7d00e1917c4d8a3a04b9f2409a9c559a2ffb35c4abe07e0" exitCode=60
Oct 31 03:15:06.869200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:06.868843    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerDied","Data":"5c941778071455f6d7d00e1917c4d8a3a04b9f2409a9c559a2ffb35c4abe07e0"}
Oct 31 03:15:06.869200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:06.869062    5831 scope.go:122] "RemoveContainer" containerID="5c941778071455f6d7d00e1917c4d8a3a04b9f2409a9c559a2ffb35c4abe07e0"
Oct 31 03:15:07.877112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:07.877074    5831 generic.go:356] "Generic (PLEG): container finished" podID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerID="29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7" exitCode=60
Oct 31 03:15:07.877510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:07.877164    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerDied","Data":"29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7"}
Oct 31 03:15:07.877510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:07.877225    5831 scope.go:122] "RemoveContainer" containerID="5c941778071455f6d7d00e1917c4d8a3a04b9f2409a9c559a2ffb35c4abe07e0"
Oct 31 03:15:07.877510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:07.877450    5831 scope.go:122] "RemoveContainer" containerID="29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7"
Oct 31 03:15:07.877788 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:15:07.877769    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-lb9np_odh-ai-gateway-infra(0b0776cf-14b4-4a97-9cb9-b0c66e581fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5"
Oct 31 03:15:08.884832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:08.884789    5831 scope.go:122] "RemoveContainer" containerID="29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7"
Oct 31 03:15:08.885245 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:15:08.885037    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-lb9np_odh-ai-gateway-infra(0b0776cf-14b4-4a97-9cb9-b0c66e581fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5"
Oct 31 03:15:12.977634 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:15:12.977594    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:15:20.130574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:20.130511    5831 scope.go:122] "RemoveContainer" containerID="29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7"
Oct 31 03:15:20.953817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:20.953784    5831 generic.go:356] "Generic (PLEG): container finished" podID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerID="5e81bb38659ec6699e31fd837725bd2a6e8056f4a7c2b34b4781ad0bedf18a5c" exitCode=60
Oct 31 03:15:20.953996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:20.953848    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerDied","Data":"5e81bb38659ec6699e31fd837725bd2a6e8056f4a7c2b34b4781ad0bedf18a5c"}
Oct 31 03:15:20.953996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:20.953905    5831 scope.go:122] "RemoveContainer" containerID="29207bccb259de7da448c5b06ff6d13c608fc93f4980a0912be33576ceccd2d7"
Oct 31 03:15:20.954326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:20.954311    5831 scope.go:122] "RemoveContainer" containerID="5e81bb38659ec6699e31fd837725bd2a6e8056f4a7c2b34b4781ad0bedf18a5c"
Oct 31 03:15:20.954605 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:15:20.954582    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890275-lb9np_odh-ai-gateway-infra(0b0776cf-14b4-4a97-9cb9-b0c66e581fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5"
Oct 31 03:15:22.099703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.099664    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:22.239647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.239619    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:22.349395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.349364    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\" (UniqueName: \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\") pod \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") "
Oct 31 03:15:22.349634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.349420    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\") pod \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\" (UID: \"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5\") "
Oct 31 03:15:22.349964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.349885    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca" pod "0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" (UID: "0b0776cf-14b4-4a97-9cb9-b0c66e581fd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:15:22.352224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.352192    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4" pod "0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" (UID: "0b0776cf-14b4-4a97-9cb9-b0c66e581fd5"). InnerVolumeSpecName "kube-api-access-6g9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:15:22.450326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.450292    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9t4\" (UniqueName: \"kubernetes.io/projected/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-kube-api-access-6g9t4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:15:22.450326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.450324    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:15:22.967333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.967296    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np" event={"ID":"0b0776cf-14b4-4a97-9cb9-b0c66e581fd5","Type":"ContainerDied","Data":"3d30dfe4bd5394dd84afa7ca116174754c157499d7775184be9dcf8bef7a656a"}
Oct 31 03:15:22.967333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.967323    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"
Oct 31 03:15:22.967642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:22.967386    5831 scope.go:122] "RemoveContainer" containerID="5e81bb38659ec6699e31fd837725bd2a6e8056f4a7c2b34b4781ad0bedf18a5c"
Oct 31 03:15:23.120331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:23.120301    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:23.166187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:23.166152    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890275-lb9np"]
Oct 31 03:15:24.135539 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:24.135499    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" path="/var/lib/kubelet/pods/0b0776cf-14b4-4a97-9cb9-b0c66e581fd5/volumes"
Oct 31 03:15:42.042312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:42.042271    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:15:42.977722 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:15:42.977678    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:15:49.528191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:15:49.528157    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:16:03.050778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.050738    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"]
Oct 31 03:16:03.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052509    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052558    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052599    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052605    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052619    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052624    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052731    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.052743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.052741    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:03.055604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.055585    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.104840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.104804    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"]
Oct 31 03:16:03.120195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120150    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.120372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120207    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.120372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120296    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c71e606-ef12-4b55-8e2a-1019c02014f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.120372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120325    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx8k\" (UniqueName: \"kubernetes.io/projected/3c71e606-ef12-4b55-8e2a-1019c02014f5-kube-api-access-xnx8k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.120556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120441    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.120556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.120510    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.147146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.147106    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 03:16:03.147146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.147138    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 03:16:03.147351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.147172    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 03:16:03.149495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.149481    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-trl42\""
Oct 31 03:16:03.221896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.221857    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.221896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.221895    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.221914    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c71e606-ef12-4b55-8e2a-1019c02014f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.221932    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx8k\" (UniqueName: \"kubernetes.io/projected/3c71e606-ef12-4b55-8e2a-1019c02014f5-kube-api-access-xnx8k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.221976    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.222008    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.222362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.222403    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.222475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.222439    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.224468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.224445    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c71e606-ef12-4b55-8e2a-1019c02014f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.224663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.224646    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c71e606-ef12-4b55-8e2a-1019c02014f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.275868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.275835    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx8k\" (UniqueName: \"kubernetes.io/projected/3c71e606-ef12-4b55-8e2a-1019c02014f5-kube-api-access-xnx8k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw\" (UID: \"3c71e606-ef12-4b55-8e2a-1019c02014f5\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.368808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.368716    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:03.742371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:03.742336    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"]
Oct 31 03:16:03.746221 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:16:03.746194    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c71e606_ef12_4b55_8e2a_1019c02014f5.slice/crio-b27fd5d5b342f28281cc6f6e1d1d192873b74d03f16bd6e74537ca2421b24c5b WatchSource:0}: Error finding container b27fd5d5b342f28281cc6f6e1d1d192873b74d03f16bd6e74537ca2421b24c5b: Status 404 returned error can't find the container with id b27fd5d5b342f28281cc6f6e1d1d192873b74d03f16bd6e74537ca2421b24c5b
Oct 31 03:16:04.075400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:04.075311    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:16:04.199477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:04.199444    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw" event={"ID":"3c71e606-ef12-4b55-8e2a-1019c02014f5","Type":"ContainerStarted","Data":"b27fd5d5b342f28281cc6f6e1d1d192873b74d03f16bd6e74537ca2421b24c5b"}
Oct 31 03:16:06.213511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:06.213479    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw" event={"ID":"3c71e606-ef12-4b55-8e2a-1019c02014f5","Type":"ContainerStarted","Data":"41a5b67bcf7f982d82bd54f7db5dcc243f3b8816576ed78ad639a44bb8a2aac5"}
Oct 31 03:16:06.321480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:06.321411    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw" podStartSLOduration=1.700008724 podStartE2EDuration="3.321397513s" totalImagesPullingTime="1.621388789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:06.319307435 +0000 UTC m=+516.333869143" watchObservedRunningTime="2026-10-31 03:16:06.321397513 +0000 UTC m=+516.335959215"
Oct 31 03:16:06.369235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:06.369201    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:12.977957 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:16:12.977914    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:16:13.369375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:13.369289    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:13.382873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:13.382844    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:14.270824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:14.270793    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw"
Oct 31 03:16:19.799560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:19.799512    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:16:35.364670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.364588    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 03:16:35.365145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.365128    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0776cf-14b4-4a97-9cb9-b0c66e581fd5" containerName="cleanup"
Oct 31 03:16:35.368907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.368885    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.417976    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2lc\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418013    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418032    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418105    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418126    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418233    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418262    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418293    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.418443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.418308    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.465375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.465342    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 03:16:35.518954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.518919    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.518961    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519023    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519050    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519093    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519116    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519168    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2lc\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519200    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519226    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519387    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519472    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519596    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.519835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519803    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.520131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.519853    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.521794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.521775    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.521886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.521779    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.572222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.572187    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.572415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.572391    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2lc\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-hxx62\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:35.683367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:35.683335    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:36.146895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.146810    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 03:16:36.150149 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:16:36.150118    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca94099_d551_4e53_a9aa_785372780924.slice/crio-63174388b4b6953557f9dfbdb4d063ba063f496e055e4f0ce0f6ba3e45637417 WatchSource:0}: Error finding container 63174388b4b6953557f9dfbdb4d063ba063f496e055e4f0ce0f6ba3e45637417: Status 404 returned error can't find the container with id 63174388b4b6953557f9dfbdb4d063ba063f496e055e4f0ce0f6ba3e45637417
Oct 31 03:16:36.152286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.152254    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:16:36.152365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.152323    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:16:36.152365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.152355    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:16:36.385960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.385920    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" event={"ID":"fca94099-d551-4e53-a9aa-785372780924","Type":"ContainerStarted","Data":"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"}
Oct 31 03:16:36.386411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.385990    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" event={"ID":"fca94099-d551-4e53-a9aa-785372780924","Type":"ContainerStarted","Data":"63174388b4b6953557f9dfbdb4d063ba063f496e055e4f0ce0f6ba3e45637417"}
Oct 31 03:16:36.495371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.495315    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podStartSLOduration=1.495297455 podStartE2EDuration="1.495297455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:36.494127946 +0000 UTC m=+546.508689653" watchObservedRunningTime="2026-10-31 03:16:36.495297455 +0000 UTC m=+546.509859163"
Oct 31 03:16:36.687681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:36.687634    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:37.704665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:37.704629    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:37.705182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:37.704680    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:38.684569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:38.683894    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:38.684569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:38.683957    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:39.696552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.686624    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:39.696552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.686684    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:39.905279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.905241    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d8444996-2rgdr"]
Oct 31 03:16:39.912406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.912380    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:39.952992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.952911    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zj62\" (UniqueName: \"kubernetes.io/projected/12c09a2b-78b4-4175-826f-781805aee163-kube-api-access-4zj62\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:39.952992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.952964    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c09a2b-78b4-4175-826f-781805aee163-tls-cert\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:39.963158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:39.963121    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d8444996-2rgdr"]
Oct 31 03:16:40.005608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.005567    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bjvt\""
Oct 31 03:16:40.006722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.006697    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 03:16:40.054266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.054230    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zj62\" (UniqueName: \"kubernetes.io/projected/12c09a2b-78b4-4175-826f-781805aee163-kube-api-access-4zj62\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:40.054432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.054289    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c09a2b-78b4-4175-826f-781805aee163-tls-cert\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:40.057183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.057152    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c09a2b-78b4-4175-826f-781805aee163-tls-cert\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:40.107659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.107620    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zj62\" (UniqueName: \"kubernetes.io/projected/12c09a2b-78b4-4175-826f-781805aee163-kube-api-access-4zj62\") pod \"authorino-5d8444996-2rgdr\" (UID: \"12c09a2b-78b4-4175-826f-781805aee163\") " pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:40.225632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.225512    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8444996-2rgdr"
Oct 31 03:16:40.621105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.621081    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d8444996-2rgdr"]
Oct 31 03:16:40.624362 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:16:40.624328    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c09a2b_78b4_4175_826f_781805aee163.slice/crio-0585ea685a99ee455c25c447e5a51a39a3b35c089d49eb95eabe9e9af99cea8a WatchSource:0}: Error finding container 0585ea685a99ee455c25c447e5a51a39a3b35c089d49eb95eabe9e9af99cea8a: Status 404 returned error can't find the container with id 0585ea685a99ee455c25c447e5a51a39a3b35c089d49eb95eabe9e9af99cea8a
Oct 31 03:16:40.694113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.693970    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:40.694113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:40.694046    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:41.432059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:41.432021    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8444996-2rgdr" event={"ID":"12c09a2b-78b4-4175-826f-781805aee163","Type":"ContainerStarted","Data":"0585ea685a99ee455c25c447e5a51a39a3b35c089d49eb95eabe9e9af99cea8a"}
Oct 31 03:16:41.689626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:41.688926    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:41.689626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:41.688989    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:42.683752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:42.683722    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:42.684068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:42.683785    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:42.979007 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:16:42.978963    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:16:43.446232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:43.446195    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8444996-2rgdr" event={"ID":"12c09a2b-78b4-4175-826f-781805aee163","Type":"ContainerStarted","Data":"783ca6713a9dcb074f2beab2c2741cb5bf80db6ce54cae08a74df9db16cf5e6c"}
Oct 31 03:16:43.555091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:43.555035    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5d8444996-2rgdr" podStartSLOduration=2.571094233 podStartE2EDuration="4.555018969s" totalImagesPullingTime="1.983924736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:43.552483873 +0000 UTC m=+553.567045581" watchObservedRunningTime="2026-10-31 03:16:43.555018969 +0000 UTC m=+553.569580677"
Oct 31 03:16:43.684393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:43.684353    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:16:43.684941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:43.684424    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:16:44.688715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:44.688675    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:44.689193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:44.689067    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:16:44.689984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:16:44.689961    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 03:17:12.977093 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:17:12.977059    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:17:15.900385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.900348    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:17:15.905826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.905798    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:15.958007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.957975    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4072c5d1-openshift-default-dockercfg-ktzzt\""
Oct 31 03:17:15.999628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.999586    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:15.999816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.999647    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:15.999816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.999791    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:15.999949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.999840    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.000000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:15.999974    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jk7g\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.000103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.000081    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.000217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.000192    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.000271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.000242    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.000323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.000275    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.004694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.004669    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:17:16.101490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101451    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101501    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101574    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101669    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101699    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.101733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101725    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101757    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.101809    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jk7g\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.102136    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.102210    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.102269    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.102381    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.102476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.102422    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.104396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.104373    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.104642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.104625    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.156732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.156657    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.156732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.156700    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jk7g\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\") pod \"e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") " pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.220231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.220192    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:16.515103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.515069    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:17:16.519898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.519861    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.568730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.568693    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bac33a89-gw-openshift-default-dockercfg-rwsc7\""
Oct 31 03:17:16.575520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.575489    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:17:16.606923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.606880    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.606931    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607003    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607114    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607153    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607225    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607265    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607337    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.607408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.607376    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhngc\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.677980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.677945    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:17:16.682019 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:16.681990    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f281655_8ff0_4542_98ae_8bb67c8bcb7d.slice/crio-aa7497f4f42f81afe22d684ee65a50a2f691ca085705e361bae81dd270793c78 WatchSource:0}: Error finding container aa7497f4f42f81afe22d684ee65a50a2f691ca085705e361bae81dd270793c78: Status 404 returned error can't find the container with id aa7497f4f42f81afe22d684ee65a50a2f691ca085705e361bae81dd270793c78
Oct 31 03:17:16.684657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.684627    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:16.684720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.684693    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:16.684755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.684721    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:16.708397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708372    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708412    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708501    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708598    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708638    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhngc\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708687    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708723    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.708944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708760    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.709003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.708964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.709062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.709044    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.709160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.709139    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.709219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.709199    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.709285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.709265    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.711000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.710977    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.711419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.711399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.764674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.764641    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.764801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.764705    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhngc\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\") pod \"e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") " pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:16.835075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:16.834990    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:17.258000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.257973    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:17:17.262850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.262815    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:17.262967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.262896    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:17.262967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.262950    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:17.642295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.642189    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerStarted","Data":"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"}
Oct 31 03:17:17.642295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.642266    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerStarted","Data":"aa7497f4f42f81afe22d684ee65a50a2f691ca085705e361bae81dd270793c78"}
Oct 31 03:17:17.643972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.643946    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerStarted","Data":"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"}
Oct 31 03:17:17.644124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.643995    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerStarted","Data":"98a74fa72df0cfab22794d80b0ac36a018de66549589cc24c26496a43bae2dea"}
Oct 31 03:17:17.835588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.835523    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:17.904467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.904361    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podStartSLOduration=2.9043399340000002 podStartE2EDuration="2.904339934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:17.791037216 +0000 UTC m=+587.805598925" watchObservedRunningTime="2026-10-31 03:17:17.904339934 +0000 UTC m=+587.918901634"
Oct 31 03:17:17.905716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:17.905682    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podStartSLOduration=1.9056707670000002 podStartE2EDuration="1.905670767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:17.902875277 +0000 UTC m=+587.917437004" watchObservedRunningTime="2026-10-31 03:17:17.905670767 +0000 UTC m=+587.920232475"
Oct 31 03:17:18.109253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.109216    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:17:18.113788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.113766    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144284    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144336    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144362    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144397    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144432    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144471    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144500    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9r7\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144572    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.149550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.144634    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.164159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.164072    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-46d24f95-openshift-default-dockercfg-zjnlc\""
Oct 31 03:17:18.175131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.175099    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:17:18.220808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.220769    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:18.225701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.225675    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:18.245796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245755    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245803    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245824    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245854    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245879    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245917    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.245982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.245947    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9r7\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246037    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246344    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246411    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246577    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246604    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.246887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.246864    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.247158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.247136    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.248747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.248728    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.249040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.249021    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.304310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.304272    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.305740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.305715    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9r7\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\") pod \"e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") " pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.428782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.428730    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:18.650036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.650007    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:18.650884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.650863    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:18.841468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.841353    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:18.863798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.863765    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:17:18.867408 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:18.867368    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f5156a_c641_4ec4_aa19_3a504880b23c.slice/crio-4f48877e61d59ddeb35ed172d41b037661048ac397b484f868344a3826d13c95 WatchSource:0}: Error finding container 4f48877e61d59ddeb35ed172d41b037661048ac397b484f868344a3826d13c95: Status 404 returned error can't find the container with id 4f48877e61d59ddeb35ed172d41b037661048ac397b484f868344a3826d13c95
Oct 31 03:17:18.869797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.869764    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:18.869880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.869828    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:18.869880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:18.869857    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:19.656756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:19.656724    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerStarted","Data":"71e5826b0152cd9191e9c1a474f72ae35adc1b049e37b53e23677c3940cbd771"}
Oct 31 03:17:19.657204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:19.656777    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerStarted","Data":"4f48877e61d59ddeb35ed172d41b037661048ac397b484f868344a3826d13c95"}
Oct 31 03:17:19.657274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:19.657227    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:19.657894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:19.657876    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:19.771998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:19.771913    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podStartSLOduration=1.771888835 podStartE2EDuration="1.771888835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:19.770947651 +0000 UTC m=+589.785509359" watchObservedRunningTime="2026-10-31 03:17:19.771888835 +0000 UTC m=+589.786450540"
Oct 31 03:17:20.429278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:20.429243    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:20.435428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:20.435402    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:20.663939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:20.663910    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:20.665248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:20.665230    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:17:28.920074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:28.920031    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:17:28.923986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:28.923963    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:28.974295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:28.974246    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:28.974509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:28.974355    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mtg\" (UniqueName: \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.019938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.019902    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2f427162\""
Oct 31 03:17:29.020144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.020049    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2f427162-dockercfg-9vl9p\""
Oct 31 03:17:29.023626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.023593    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:17:29.075934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.075896    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mtg\" (UniqueName: \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.076106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.076018    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.076742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.076724    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.077013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.076994    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:17:29.080816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.080790    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.129902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.129874    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mtg\" (UniqueName: \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\") pod \"payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.133996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.133971    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:17:29.177313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.177230    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.177476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.177370    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgrp\" (UniqueName: \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.240349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.240309    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:29.278550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.278490    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.278749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.278584    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgrp\" (UniqueName: \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.279649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.279622    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.335919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.334621    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgrp\" (UniqueName: \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\") pod \"payload-processing-e2e-ait-2f427162-574d49d87-nbrpt\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") " pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.394953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.394916    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:29.630845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.630817    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:17:29.633134 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:29.633102    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069ee20a_62fe_408b_97ed_9392074a2c3d.slice/crio-1c2e37a789ecf9167d2d14f5c4b3d8f1dfc4c2fb0fe15c45e267c457db5f2e50 WatchSource:0}: Error finding container 1c2e37a789ecf9167d2d14f5c4b3d8f1dfc4c2fb0fe15c45e267c457db5f2e50: Status 404 returned error can't find the container with id 1c2e37a789ecf9167d2d14f5c4b3d8f1dfc4c2fb0fe15c45e267c457db5f2e50
Oct 31 03:17:29.728484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.728393    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" event={"ID":"069ee20a-62fe-408b-97ed-9392074a2c3d","Type":"ContainerStarted","Data":"1c2e37a789ecf9167d2d14f5c4b3d8f1dfc4c2fb0fe15c45e267c457db5f2e50"}
Oct 31 03:17:29.770903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:29.770858    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:17:29.773220 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:29.773191    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5761a401_6791_494c_a37d_eb1d22c94ece.slice/crio-faa3396b7f6bb5c016ed2c2d4f51c488ae9f68bcd249ffc3294e0082423548aa WatchSource:0}: Error finding container faa3396b7f6bb5c016ed2c2d4f51c488ae9f68bcd249ffc3294e0082423548aa: Status 404 returned error can't find the container with id faa3396b7f6bb5c016ed2c2d4f51c488ae9f68bcd249ffc3294e0082423548aa
Oct 31 03:17:30.202816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.202787    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:17:30.204073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.204047    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:17:30.213351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.213328    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:17:30.214503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.214482    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:17:30.217055 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.217037    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:17:30.217981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.217963    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:17:30.736772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:30.736733    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" event={"ID":"5761a401-6791-494c-a37d-eb1d22c94ece","Type":"ContainerStarted","Data":"faa3396b7f6bb5c016ed2c2d4f51c488ae9f68bcd249ffc3294e0082423548aa"}
Oct 31 03:17:31.443614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.443580    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:17:31.448091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.448006    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.503845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.503810    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4072c5d1\""
Oct 31 03:17:31.509239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.509208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhtq\" (UniqueName: \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.509388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.509266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.553777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.553741    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4072c5d1-dockercfg-kmr7z\""
Oct 31 03:17:31.553999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.553960    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:17:31.610794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.610756    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhtq\" (UniqueName: \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.611007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.610831    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.611691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.611654    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.665302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.665270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhtq\" (UniqueName: \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\") pod \"payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") " pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:31.767762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:31.767650    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:32.167604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:32.167489    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:17:32.171996 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:32.171958    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe92e750_da95_456a_8246_577353ed5f70.slice/crio-3deb92302c1a2f66b529ea9bb9e04b2a53188a2f6635ae2a36eeb550fc1eb575 WatchSource:0}: Error finding container 3deb92302c1a2f66b529ea9bb9e04b2a53188a2f6635ae2a36eeb550fc1eb575: Status 404 returned error can't find the container with id 3deb92302c1a2f66b529ea9bb9e04b2a53188a2f6635ae2a36eeb550fc1eb575
Oct 31 03:17:32.774153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:32.774110    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" event={"ID":"be92e750-da95-456a-8246-577353ed5f70","Type":"ContainerStarted","Data":"3deb92302c1a2f66b529ea9bb9e04b2a53188a2f6635ae2a36eeb550fc1eb575"}
Oct 31 03:17:33.822789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:33.821844    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" event={"ID":"069ee20a-62fe-408b-97ed-9392074a2c3d","Type":"ContainerStarted","Data":"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"}
Oct 31 03:17:33.822789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:33.822629    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:33.825462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:33.825433    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" event={"ID":"5761a401-6791-494c-a37d-eb1d22c94ece","Type":"ContainerStarted","Data":"6122e49c31b66281ff9b88a53c19c0949cdd8e2c57eb0657532aa0ff6b767e95"}
Oct 31 03:17:33.825585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:33.825518    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:33.930132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:33.930079    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" podStartSLOduration=2.280801543 podStartE2EDuration="5.930061717s" totalImagesPullingTime="3.649260174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:33.9282982 +0000 UTC m=+603.942859908" watchObservedRunningTime="2026-10-31 03:17:33.930061717 +0000 UTC m=+603.944623425"
Oct 31 03:17:34.034457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:34.034398    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" podStartSLOduration=1.6017284360000001 podStartE2EDuration="5.034378942s" totalImagesPullingTime="3.432650506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:34.032500962 +0000 UTC m=+604.047062669" watchObservedRunningTime="2026-10-31 03:17:34.034378942 +0000 UTC m=+604.048940654"
Oct 31 03:17:34.835859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:34.835821    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" event={"ID":"be92e750-da95-456a-8246-577353ed5f70","Type":"ContainerStarted","Data":"4c1f5fba3510b27f66de6574ea7ad7b5a8cac6e0f26b825103060a080c9206ad"}
Oct 31 03:17:34.943285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:34.943229    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" podStartSLOduration=1.894149805 podStartE2EDuration="3.943213701s" totalImagesPullingTime="2.049063896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:34.941893197 +0000 UTC m=+604.956454905" watchObservedRunningTime="2026-10-31 03:17:34.943213701 +0000 UTC m=+604.957775408"
Oct 31 03:17:35.843731 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:35.843700    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:37.979147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:37.979115    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:17:42.619555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.619335    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:17:42.640285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.640240    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:42.656276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.655547    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:17:42.694090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.692942    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15000fe0\""
Oct 31 03:17:42.697203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.697145    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15000fe0-dockercfg-64frl\""
Oct 31 03:17:42.746980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.744784    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:17:42.750230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.750206    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:42.774798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.774759    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nhp\" (UniqueName: \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:42.775123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.775099    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:42.808807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.808766    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:17:42.887823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.887729    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nhp\" (UniqueName: \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:42.888092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.888070    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcvc\" (UniqueName: \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:42.888345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.888326    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:42.893423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.893400    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:42.912927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:42.893329    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:43.012557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.010613    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nhp\" (UniqueName: \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\") pod \"payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:43.016554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.013911    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcvc\" (UniqueName: \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.016554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.013999    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.016554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.014937    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.038684 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:17:43.038544    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:17:43.068327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.068292    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcvc\" (UniqueName: \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\") pod \"payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") " pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.070185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.070159    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.256018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.255917    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:43.532465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.532252    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:17:43.533306 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:17:43.533272    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4509f7c8_881e_4deb_b9e7_3ded6e6930d8.slice/crio-f430a8a0be9afb5c0d652f99c3b0bcced8d0f2679e17a767408bf69e7e7f0fe2 WatchSource:0}: Error finding container f430a8a0be9afb5c0d652f99c3b0bcced8d0f2679e17a767408bf69e7e7f0fe2: Status 404 returned error can't find the container with id f430a8a0be9afb5c0d652f99c3b0bcced8d0f2679e17a767408bf69e7e7f0fe2
Oct 31 03:17:43.731402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.731325    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:17:43.972519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.972486    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" event={"ID":"4509f7c8-881e-4deb-b9e7-3ded6e6930d8","Type":"ContainerStarted","Data":"0da3e404e1bc1ea25b8060d2581fe395f223043edea40aae7482d923dae07f9b"}
Oct 31 03:17:43.972743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.972567    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:43.972743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.972589    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" event={"ID":"4509f7c8-881e-4deb-b9e7-3ded6e6930d8","Type":"ContainerStarted","Data":"f430a8a0be9afb5c0d652f99c3b0bcced8d0f2679e17a767408bf69e7e7f0fe2"}
Oct 31 03:17:43.974383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.974360    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" event={"ID":"8bbccfca-df17-4416-b174-a634e81b0254","Type":"ContainerStarted","Data":"d4f04ddf84a884c54ce04cadf11aaa539951ffcb6d7374dccbba6b56d5c20234"}
Oct 31 03:17:43.974496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.974397    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" event={"ID":"8bbccfca-df17-4416-b174-a634e81b0254","Type":"ContainerStarted","Data":"ead635dd8042e71ec11f803abc095a21b379085cad1691d44b0da3debb13486c"}
Oct 31 03:17:43.974578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:43.974563    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:44.090641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:44.085188    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" podStartSLOduration=2.085170299 podStartE2EDuration="2.085170299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:44.083002703 +0000 UTC m=+614.097564436" watchObservedRunningTime="2026-10-31 03:17:44.085170299 +0000 UTC m=+614.099732008"
Oct 31 03:17:44.202934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:44.202862    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" podStartSLOduration=2.202826471 podStartE2EDuration="2.202826471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:44.200094216 +0000 UTC m=+614.214655927" watchObservedRunningTime="2026-10-31 03:17:44.202826471 +0000 UTC m=+614.217388180"
Oct 31 03:17:44.840115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:44.840018    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:17:45.847522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:45.847490    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:17:46.854315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:46.854285    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:17:48.009742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.009706    5831 generic.go:356] "Generic (PLEG): container finished" podID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerID="75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48" exitCode=0
Oct 31 03:17:48.010307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.009780    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerDied","Data":"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"}
Oct 31 03:17:48.010307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.010217    5831 scope.go:122] "RemoveContainer" containerID="75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"
Oct 31 03:17:48.011783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.011763    5831 generic.go:356] "Generic (PLEG): container finished" podID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerID="f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f" exitCode=0
Oct 31 03:17:48.011897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.011815    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.011897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.011838    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerDied","Data":"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"}
Oct 31 03:17:48.012007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.011903    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.012007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.011950    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.012171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.012156    5831 scope.go:122] "RemoveContainer" containerID="f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"
Oct 31 03:17:48.013384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.013358    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.013451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.013435    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.013499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.013484    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:17:48.649921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:48.649880    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:49.033343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.033307    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerStarted","Data":"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"}
Oct 31 03:17:49.037523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.037484    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerStarted","Data":"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"}
Oct 31 03:17:49.229985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.229947    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:49.232673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.232639    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:49.232942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.232913    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:49.657803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.657762    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:49.839871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.839836    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:49.840274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.840252    5831 patch_prober.go:29] interesting pod/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:17:49.840477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:49.840453    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 03:17:50.230011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:50.229971    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:50.230662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:50.230629    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:50.836000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:50.835951    5831 patch_prober.go:29] interesting pod/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:17:50.836211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:50.836012    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 03:17:51.222555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:51.221954    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:51.222555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:51.222019    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:51.836376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:51.836043    5831 patch_prober.go:29] interesting pod/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:17:51.836376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:51.836094    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 03:17:52.221435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:52.220948    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:52.221435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:52.221031    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:52.836093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:52.836046    5831 patch_prober.go:29] interesting pod/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:17:52.836321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:52.836112    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 03:17:53.230066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:53.230018    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:53.230618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:53.230148    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:53.835924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:53.835881    5831 patch_prober.go:29] interesting pod/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:17:53.836112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:53.835947    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 03:17:54.221345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.221309    5831 patch_prober.go:29] interesting pod/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:17:54.221516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.221384    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:17:54.840499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.840468    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:54.841498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.841478    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:17:54.986391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.986360    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:17:54.986599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:54.986436    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:17:55.224831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:55.224801    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:55.225218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:55.225201    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:55.225884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:55.225865    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:17:58.671862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:17:58.671769    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:18:06.481372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.481338    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:18:06.625442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.625406    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"]
Oct 31 03:18:06.629649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.629616    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.724883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.724848    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:18:06.725091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.724900    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:18:06.725091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.724924    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:18:06.725091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.724946    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:18:06.727504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.727478    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"]
Oct 31 03:18:06.798600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.798491    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rjj\" (UniqueName: \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.798760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.798623    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.798760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.798684    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.899989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.899947    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rjj\" (UniqueName: \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.900190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.900001    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.900190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.900057    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.900686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.900663    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.902585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.902557    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:06.953906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:06.953867    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rjj\" (UniqueName: \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\") pod \"maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:07.244177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:07.244141    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:07.636606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:07.636423    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"]
Oct 31 03:18:07.640125 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:07.640093    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eeedc7_4919_4e03_9c86_a843c18e8ecc.slice/crio-ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98 WatchSource:0}: Error finding container ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98: Status 404 returned error can't find the container with id ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98
Oct 31 03:18:08.165152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:08.165111    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g" event={"ID":"30eeedc7-4919-4e03-9c86-a843c18e8ecc","Type":"ContainerStarted","Data":"45baf147f06e41ba02f741c7600cf369f0e52380a9ce08a8447ad74fd53a287b"}
Oct 31 03:18:08.165359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:08.165185    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g" event={"ID":"30eeedc7-4919-4e03-9c86-a843c18e8ecc","Type":"ContainerStarted","Data":"ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98"}
Oct 31 03:18:08.271725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:08.271604    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g" podStartSLOduration=2.271588954 podStartE2EDuration="2.271588954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:08.270580529 +0000 UTC m=+638.285142238" watchObservedRunningTime="2026-10-31 03:18:08.271588954 +0000 UTC m=+638.286150661"
Oct 31 03:18:09.173354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:09.173258    5831 generic.go:356] "Generic (PLEG): container finished" podID="30eeedc7-4919-4e03-9c86-a843c18e8ecc" containerID="45baf147f06e41ba02f741c7600cf369f0e52380a9ce08a8447ad74fd53a287b" exitCode=0
Oct 31 03:18:09.173354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:09.173303    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g" event={"ID":"30eeedc7-4919-4e03-9c86-a843c18e8ecc","Type":"ContainerDied","Data":"45baf147f06e41ba02f741c7600cf369f0e52380a9ce08a8447ad74fd53a287b"}
Oct 31 03:18:10.411446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.411415    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:10.539346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.539259    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\" (UniqueName: \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\") pod \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") "
Oct 31 03:18:10.539812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.539778    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\") pod \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") "
Oct 31 03:18:10.539915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.539837    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\") pod \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\" (UID: \"30eeedc7-4919-4e03-9c86-a843c18e8ecc\") "
Oct 31 03:18:10.540115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.540060    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca" pod "30eeedc7-4919-4e03-9c86-a843c18e8ecc" (UID: "30eeedc7-4919-4e03-9c86-a843c18e8ecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:10.540580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.540212    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:10.541792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.541767    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj" pod "30eeedc7-4919-4e03-9c86-a843c18e8ecc" (UID: "30eeedc7-4919-4e03-9c86-a843c18e8ecc"). InnerVolumeSpecName "kube-api-access-s8rjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:10.542849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.542828    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca" pod "30eeedc7-4919-4e03-9c86-a843c18e8ecc" (UID: "30eeedc7-4919-4e03-9c86-a843c18e8ecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:10.640929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.640889    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rjj\" (UniqueName: \"kubernetes.io/projected/30eeedc7-4919-4e03-9c86-a843c18e8ecc-kube-api-access-s8rjj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:10.640929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:10.640924    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eeedc7-4919-4e03-9c86-a843c18e8ecc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:11.135349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.135316    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"]
Oct 31 03:18:11.135806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.135793    5831 cpu_manager.go:414] "containerMap: removing container" podUID="30eeedc7-4919-4e03-9c86-a843c18e8ecc" containerName="revoke-keys"
Oct 31 03:18:11.135859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.135807    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30eeedc7-4919-4e03-9c86-a843c18e8ecc" containerName="revoke-keys"
Oct 31 03:18:11.135918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.135909    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="30eeedc7-4919-4e03-9c86-a843c18e8ecc" containerName="revoke-keys"
Oct 31 03:18:11.139296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.139276    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.211237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.211204    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"]
Oct 31 03:18:11.213271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.213247    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"
Oct 31 03:18:11.213422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.213330    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g" event={"ID":"30eeedc7-4919-4e03-9c86-a843c18e8ecc","Type":"ContainerDied","Data":"ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98"}
Oct 31 03:18:11.213422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.213388    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae63b99d71a05dc3bc14a4a14b16cc8ebb231074f8488482b72716c9a93cab98"
Oct 31 03:18:11.246831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.246800    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55xw\" (UniqueName: \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.246985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.246918    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.246985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.246959    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.348373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.348334    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55xw\" (UniqueName: \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.348589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.348422    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.348589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.348452    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.349219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.349190    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.351065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.351037    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.406065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.405986    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55xw\" (UniqueName: \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\") pod \"maas-api-revoke-keys-e2e-migrate-bac33a89-phq45\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.452860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.452826    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:11.850492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:11.850443    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"]
Oct 31 03:18:11.855323 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:11.855292    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ce9679_c479_4bf1_9f0c_520d9e09fd75.slice/crio-c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519 WatchSource:0}: Error finding container c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519: Status 404 returned error can't find the container with id c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519
Oct 31 03:18:12.223751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:12.223703    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45" event={"ID":"f6ce9679-c479-4bf1-9f0c-520d9e09fd75","Type":"ContainerStarted","Data":"c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519"}
Oct 31 03:18:12.985258 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:12.985220    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:18:13.232778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:13.232745    5831 generic.go:356] "Generic (PLEG): container finished" podID="f6ce9679-c479-4bf1-9f0c-520d9e09fd75" containerID="cb178313a9173960be2e8aefb2e384237e12bcd6974dd5e8a8f514b4bddab941" exitCode=0
Oct 31 03:18:13.233002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:13.232878    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45" event={"ID":"f6ce9679-c479-4bf1-9f0c-520d9e09fd75","Type":"ContainerDied","Data":"cb178313a9173960be2e8aefb2e384237e12bcd6974dd5e8a8f514b4bddab941"}
Oct 31 03:18:14.405883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.405857    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:14.479326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.479291    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\" (UniqueName: \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\") pod \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") "
Oct 31 03:18:14.479613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.479368    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\") pod \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") "
Oct 31 03:18:14.479613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.479457    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\") pod \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\" (UID: \"f6ce9679-c479-4bf1-9f0c-520d9e09fd75\") "
Oct 31 03:18:14.479844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.479817    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca" pod "f6ce9679-c479-4bf1-9f0c-520d9e09fd75" (UID: "f6ce9679-c479-4bf1-9f0c-520d9e09fd75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:14.482227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.482199    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca" pod "f6ce9679-c479-4bf1-9f0c-520d9e09fd75" (UID: "f6ce9679-c479-4bf1-9f0c-520d9e09fd75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:14.482362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.482201    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw" pod "f6ce9679-c479-4bf1-9f0c-520d9e09fd75" (UID: "f6ce9679-c479-4bf1-9f0c-520d9e09fd75"). InnerVolumeSpecName "kube-api-access-r55xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:14.581150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.581067    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:14.581150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.581100    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55xw\" (UniqueName: \"kubernetes.io/projected/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-kube-api-access-r55xw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:14.581150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:14.581111    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ce9679-c479-4bf1-9f0c-520d9e09fd75-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:15.248486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:15.248446    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45" event={"ID":"f6ce9679-c479-4bf1-9f0c-520d9e09fd75","Type":"ContainerDied","Data":"c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519"}
Oct 31 03:18:15.248686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:15.248500    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c56962609fdee146761afc951488d5123959f901636f75fa54687ae4f4606519"
Oct 31 03:18:15.248686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:15.248464    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"
Oct 31 03:18:16.256741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.256708    5831 generic.go:356] "Generic (PLEG): container finished" podID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerID="71e5826b0152cd9191e9c1a474f72ae35adc1b049e37b53e23677c3940cbd771" exitCode=0
Oct 31 03:18:16.257191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.256786    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerDied","Data":"71e5826b0152cd9191e9c1a474f72ae35adc1b049e37b53e23677c3940cbd771"}
Oct 31 03:18:16.257191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.257171    5831 scope.go:122] "RemoveContainer" containerID="71e5826b0152cd9191e9c1a474f72ae35adc1b049e37b53e23677c3940cbd771"
Oct 31 03:18:16.258432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.258407    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:16.258493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.258473    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:16.258555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:16.258523    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:17.265606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:17.265571    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerStarted","Data":"09ab551ac88d3bf73b828e6d76a219a00dba3f0f4a7667c31c5f06d098288470"}
Oct 31 03:18:17.429611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:17.429572    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:18:17.431022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:17.430994    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:18:17.431182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:17.431045    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 03:18:18.204064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:18.204012    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:18:18.429727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:18.429690    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:18:18.430167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:18.429746    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 03:18:19.429700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:19.429662    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:18:19.429884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:19.429716    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 03:18:20.430414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:20.430374    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:18:20.430948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:20.430433    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 03:18:20.664224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:20.664190    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:18:21.430393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:21.430364    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:18:21.430585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:21.430417    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 03:18:22.066269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.066238    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"]
Oct 31 03:18:22.066753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.066739    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f6ce9679-c479-4bf1-9f0c-520d9e09fd75" containerName="revoke-keys"
Oct 31 03:18:22.066806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.066754    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ce9679-c479-4bf1-9f0c-520d9e09fd75" containerName="revoke-keys"
Oct 31 03:18:22.066846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.066835    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ce9679-c479-4bf1-9f0c-520d9e09fd75" containerName="revoke-keys"
Oct 31 03:18:22.070436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.070415    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.121045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.121005    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:18:22.166349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166307    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl5l\" (UniqueName: \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.166600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166400    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.166600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166452    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.166600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166456    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:18:22.166600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166456    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:18:22.166600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.166478    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:18:22.168978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.168959    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"]
Oct 31 03:18:22.267336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.267299    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.267555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.267403    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl5l\" (UniqueName: \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.267555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.267459    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.268195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.268172    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.270218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.270195    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.321600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.321495    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl5l\" (UniqueName: \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\") pod \"maas-api-revoke-keys-e2e-ait-2f427162-k2lg8\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.405424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.405377    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:22.515589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.515507    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:18:22.518265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.518237    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:18:22.784270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:22.784237    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"]
Oct 31 03:18:22.785350 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:22.785321    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a5c52e_7ce5_41bc_bd06_364dbd296fba.slice/crio-13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb WatchSource:0}: Error finding container 13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb: Status 404 returned error can't find the container with id 13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb
Oct 31 03:18:23.205132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.205091    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" podUID="be92e750-da95-456a-8246-577353ed5f70" containerName="payload-processing" containerID="cri-o://4c1f5fba3510b27f66de6574ea7ad7b5a8cac6e0f26b825103060a080c9206ad" gracePeriod=25
Oct 31 03:18:23.324786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.324746    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8" event={"ID":"c8a5c52e-7ce5-41bc-bd06-364dbd296fba","Type":"ContainerStarted","Data":"6d8dd39f8e05f80f67031272d206ab018782e476237584a5078b228a6619f235"}
Oct 31 03:18:23.324942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.324811    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8" event={"ID":"c8a5c52e-7ce5-41bc-bd06-364dbd296fba","Type":"ContainerStarted","Data":"13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb"}
Oct 31 03:18:23.326494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.326470    5831 generic.go:356] "Generic (PLEG): container finished" podID="be92e750-da95-456a-8246-577353ed5f70" containerID="4c1f5fba3510b27f66de6574ea7ad7b5a8cac6e0f26b825103060a080c9206ad" exitCode=0
Oct 31 03:18:23.326643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.326574    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" event={"ID":"be92e750-da95-456a-8246-577353ed5f70","Type":"ContainerDied","Data":"4c1f5fba3510b27f66de6574ea7ad7b5a8cac6e0f26b825103060a080c9206ad"}
Oct 31 03:18:23.433922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.433851    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8" podStartSLOduration=1.433831215 podStartE2EDuration="1.433831215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:23.42946658 +0000 UTC m=+653.444028301" watchObservedRunningTime="2026-10-31 03:18:23.433831215 +0000 UTC m=+653.448392924"
Oct 31 03:18:23.474229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.474203    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:18:23.582000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.581968    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\" (UniqueName: \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\") pod \"be92e750-da95-456a-8246-577353ed5f70\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") "
Oct 31 03:18:23.582644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.582621    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\") pod \"be92e750-da95-456a-8246-577353ed5f70\" (UID: \"be92e750-da95-456a-8246-577353ed5f70\") "
Oct 31 03:18:23.583100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.583072    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume" pod "be92e750-da95-456a-8246-577353ed5f70" (UID: "be92e750-da95-456a-8246-577353ed5f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:23.583245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.583222    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be92e750-da95-456a-8246-577353ed5f70-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:23.586294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.586265    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq" pod "be92e750-da95-456a-8246-577353ed5f70" (UID: "be92e750-da95-456a-8246-577353ed5f70"). InnerVolumeSpecName "kube-api-access-cxhtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:23.686317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:23.686282    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhtq\" (UniqueName: \"kubernetes.io/projected/be92e750-da95-456a-8246-577353ed5f70-kube-api-access-cxhtq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:24.334417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.334383    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"
Oct 31 03:18:24.334417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.334412    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd" event={"ID":"be92e750-da95-456a-8246-577353ed5f70","Type":"ContainerDied","Data":"3deb92302c1a2f66b529ea9bb9e04b2a53188a2f6635ae2a36eeb550fc1eb575"}
Oct 31 03:18:24.334676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.334478    5831 scope.go:122] "RemoveContainer" containerID="4c1f5fba3510b27f66de6574ea7ad7b5a8cac6e0f26b825103060a080c9206ad"
Oct 31 03:18:24.336221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.336193    5831 generic.go:356] "Generic (PLEG): container finished" podID="c8a5c52e-7ce5-41bc-bd06-364dbd296fba" containerID="6d8dd39f8e05f80f67031272d206ab018782e476237584a5078b228a6619f235" exitCode=0
Oct 31 03:18:24.336349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.336295    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8" event={"ID":"c8a5c52e-7ce5-41bc-bd06-364dbd296fba","Type":"ContainerDied","Data":"6d8dd39f8e05f80f67031272d206ab018782e476237584a5078b228a6619f235"}
Oct 31 03:18:24.597148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.597114    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:18:24.602183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:24.602156    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4072c5d1-bc5c5b467-zhgpd"]
Oct 31 03:18:25.491721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.491693    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:25.605168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.605059    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\" (UniqueName: \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\") pod \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") "
Oct 31 03:18:25.605676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.605173    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\") pod \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") "
Oct 31 03:18:25.605676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.605230    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\") pod \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\" (UID: \"c8a5c52e-7ce5-41bc-bd06-364dbd296fba\") "
Oct 31 03:18:25.605971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.605686    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca" pod "c8a5c52e-7ce5-41bc-bd06-364dbd296fba" (UID: "c8a5c52e-7ce5-41bc-bd06-364dbd296fba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:25.607485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.607457    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l" pod "c8a5c52e-7ce5-41bc-bd06-364dbd296fba" (UID: "c8a5c52e-7ce5-41bc-bd06-364dbd296fba"). InnerVolumeSpecName "kube-api-access-ldl5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:25.607763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.607742    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca" pod "c8a5c52e-7ce5-41bc-bd06-364dbd296fba" (UID: "c8a5c52e-7ce5-41bc-bd06-364dbd296fba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:25.706192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.706160    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl5l\" (UniqueName: \"kubernetes.io/projected/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-kube-api-access-ldl5l\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:25.706192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.706190    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:25.706401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:25.706201    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a5c52e-7ce5-41bc-bd06-364dbd296fba-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:26.134581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:26.134551    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be92e750-da95-456a-8246-577353ed5f70" path="/var/lib/kubelet/pods/be92e750-da95-456a-8246-577353ed5f70/volumes"
Oct 31 03:18:26.351175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:26.351143    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"
Oct 31 03:18:26.351386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:26.351186    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8" event={"ID":"c8a5c52e-7ce5-41bc-bd06-364dbd296fba","Type":"ContainerDied","Data":"13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb"}
Oct 31 03:18:26.351386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:26.351233    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13d971c15f905a1a2a329f5f09850965e47eb5462e10a26b967bd8764cbe21bb"
Oct 31 03:18:27.827603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.827573    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"]
Oct 31 03:18:27.828063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828048    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c8a5c52e-7ce5-41bc-bd06-364dbd296fba" containerName="revoke-keys"
Oct 31 03:18:27.828119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828065    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a5c52e-7ce5-41bc-bd06-364dbd296fba" containerName="revoke-keys"
Oct 31 03:18:27.828119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828078    5831 cpu_manager.go:414] "containerMap: removing container" podUID="be92e750-da95-456a-8246-577353ed5f70" containerName="payload-processing"
Oct 31 03:18:27.828119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828083    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be92e750-da95-456a-8246-577353ed5f70" containerName="payload-processing"
Oct 31 03:18:27.828219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828156    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="be92e750-da95-456a-8246-577353ed5f70" containerName="payload-processing"
Oct 31 03:18:27.828219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.828171    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a5c52e-7ce5-41bc-bd06-364dbd296fba" containerName="revoke-keys"
Oct 31 03:18:27.832033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.832011    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:27.880727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.880696    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:18:27.926690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.926660    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:18:27.926690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.926671    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:18:27.926923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.926828    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:18:27.929263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.929240    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"]
Oct 31 03:18:27.931186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.931166    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:27.931227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.931201    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:27.931376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:27.931358    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd27w\" (UniqueName: \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.032323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.032290    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd27w\" (UniqueName: \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.032561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.032407    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.032561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.032433    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.033189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.033167    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.035035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.035017    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.085501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.085430    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd27w\" (UniqueName: \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\") pod \"maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.149870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.149837    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:28.525849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:28.525820    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"]
Oct 31 03:18:28.528066 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:28.528038    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c545aa_d97b_4fbb_8ff6_270ef46d7078.slice/crio-0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1 WatchSource:0}: Error finding container 0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1: Status 404 returned error can't find the container with id 0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1
Oct 31 03:18:29.372172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:29.372120    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9" event={"ID":"f7c545aa-d97b-4fbb-8ff6-270ef46d7078","Type":"ContainerStarted","Data":"2bc7b55a11abecbdcdc1df920cbaa7caf42fd7d14236bea2fa2c1de02de146dd"}
Oct 31 03:18:29.372697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:29.372181    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9" event={"ID":"f7c545aa-d97b-4fbb-8ff6-270ef46d7078","Type":"ContainerStarted","Data":"0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1"}
Oct 31 03:18:29.481100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:29.481047    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9" podStartSLOduration=2.4810337909999998 podStartE2EDuration="2.481033791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:29.480432493 +0000 UTC m=+659.494994203" watchObservedRunningTime="2026-10-31 03:18:29.481033791 +0000 UTC m=+659.495595498"
Oct 31 03:18:30.379982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:30.379947    5831 generic.go:356] "Generic (PLEG): container finished" podID="f7c545aa-d97b-4fbb-8ff6-270ef46d7078" containerID="2bc7b55a11abecbdcdc1df920cbaa7caf42fd7d14236bea2fa2c1de02de146dd" exitCode=0
Oct 31 03:18:30.380476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:30.380037    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9" event={"ID":"f7c545aa-d97b-4fbb-8ff6-270ef46d7078","Type":"ContainerDied","Data":"2bc7b55a11abecbdcdc1df920cbaa7caf42fd7d14236bea2fa2c1de02de146dd"}
Oct 31 03:18:31.550412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.541808    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:31.575793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.575762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\") pod \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") "
Oct 31 03:18:31.576006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.575904    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\" (UniqueName: \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\") pod \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") "
Oct 31 03:18:31.576006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.575936    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\") pod \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\" (UID: \"f7c545aa-d97b-4fbb-8ff6-270ef46d7078\") "
Oct 31 03:18:31.576348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.576323    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca" pod "f7c545aa-d97b-4fbb-8ff6-270ef46d7078" (UID: "f7c545aa-d97b-4fbb-8ff6-270ef46d7078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:31.578203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.578176    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w" pod "f7c545aa-d97b-4fbb-8ff6-270ef46d7078" (UID: "f7c545aa-d97b-4fbb-8ff6-270ef46d7078"). InnerVolumeSpecName "kube-api-access-gd27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:31.578203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.578173    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca" pod "f7c545aa-d97b-4fbb-8ff6-270ef46d7078" (UID: "f7c545aa-d97b-4fbb-8ff6-270ef46d7078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:31.677654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.677565    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.677654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.677596    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd27w\" (UniqueName: \"kubernetes.io/projected/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-kube-api-access-gd27w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.677654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.677607    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c545aa-d97b-4fbb-8ff6-270ef46d7078-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.752729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.752691    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"]
Oct 31 03:18:31.756670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.756640    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4072c5d1-6pf2g"]
Oct 31 03:18:31.814057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.814020    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:18:31.814263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:31.814239    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy" containerID="cri-o://e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726" gracePeriod=30
Oct 31 03:18:32.134757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:32.134723    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30eeedc7-4919-4e03-9c86-a843c18e8ecc" path="/var/lib/kubelet/pods/30eeedc7-4919-4e03-9c86-a843c18e8ecc/volumes"
Oct 31 03:18:32.393312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:32.393221    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"
Oct 31 03:18:32.393312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:32.393253    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9" event={"ID":"f7c545aa-d97b-4fbb-8ff6-270ef46d7078","Type":"ContainerDied","Data":"0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1"}
Oct 31 03:18:32.393312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:32.393303    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad0895c915721a90ee08938b3d2081afd18212b07566c56630ae5158528faf1"
Oct 31 03:18:33.015659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.015627    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"]
Oct 31 03:18:33.061905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.061874    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bac33a89-phq45"]
Oct 31 03:18:33.077545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.077498    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:18:33.078688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.078651    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy" containerID="cri-o://af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0" gracePeriod=30
Oct 31 03:18:33.154739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.154707    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:18:33.155188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.155175    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f7c545aa-d97b-4fbb-8ff6-270ef46d7078" containerName="revoke-keys"
Oct 31 03:18:33.155235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.155189    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c545aa-d97b-4fbb-8ff6-270ef46d7078" containerName="revoke-keys"
Oct 31 03:18:33.155275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.155266    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c545aa-d97b-4fbb-8ff6-270ef46d7078" containerName="revoke-keys"
Oct 31 03:18:33.159004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.158980    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.195030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.194883    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.195030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.194933    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.195030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.194969    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.195758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.195691    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.196136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.196048    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.196498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.196320    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.196498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.196375    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.196498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.196423    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.197855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.197743    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.208028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.207977    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 03:18:33.249247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.249214    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:18:33.298915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.298821    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.298915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.298873    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.298937    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.298958    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.298991    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.299040    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.299069    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.299470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.299403    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.300080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.300051    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.300370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.300343    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.300461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.300414    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.300832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.300812    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.301014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.300988    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.301186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.301058    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.320056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.315083    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.320056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.315717    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.352417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.352384    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.352607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.352398    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.473049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.473013    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:33.892775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.892605    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:18:33.895226 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:33.895200    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc49a73_12ea_40a4_afbc_08311509dd58.slice/crio-c1db455f7656a14e1b74b2454a66503610fb77efa784eb632c6d7dc88f36a71e WatchSource:0}: Error finding container c1db455f7656a14e1b74b2454a66503610fb77efa784eb632c6d7dc88f36a71e: Status 404 returned error can't find the container with id c1db455f7656a14e1b74b2454a66503610fb77efa784eb632c6d7dc88f36a71e
Oct 31 03:18:33.897501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.897463    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:33.897601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.897543    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:33.897601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:33.897575    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:34.136078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.136043    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ce9679-c479-4bf1-9f0c-520d9e09fd75" path="/var/lib/kubelet/pods/f6ce9679-c479-4bf1-9f0c-520d9e09fd75/volumes"
Oct 31 03:18:34.321059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.320598    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:18:34.322354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.322324    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"]
Oct 31 03:18:34.326586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.326560    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.374664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.374631    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:18:34.374884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.374824    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:18:34.374974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.374913    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:18:34.374974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.374939    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:18:34.417757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.417721    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:18:34.418775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.418743    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.419008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.418975    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrlh\" (UniqueName: \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.419142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.419032    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.421786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.421741    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"]
Oct 31 03:18:34.424801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.424772    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerStarted","Data":"dfad3a5cce56281a765506388af29575945554fd956b6f036236f01090f01563"}
Oct 31 03:18:34.424915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.424828    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerStarted","Data":"c1db455f7656a14e1b74b2454a66503610fb77efa784eb632c6d7dc88f36a71e"}
Oct 31 03:18:34.473283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.473255    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:34.479001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.478967    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:34.520645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.520608    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.521053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.520664    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrlh\" (UniqueName: \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.521053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.520704    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.521507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.521480    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.524336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.524305    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.582687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.582604    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrlh\" (UniqueName: \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\") pod \"maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.644260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.644231    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:34.763643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:34.762721    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podStartSLOduration=1.7627006619999999 podStartE2EDuration="1.762700662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:34.760836524 +0000 UTC m=+664.775398232" watchObservedRunningTime="2026-10-31 03:18:34.762700662 +0000 UTC m=+664.777262372"
Oct 31 03:18:35.031505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.031476    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"]
Oct 31 03:18:35.033781 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:35.033751    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fa838b_3e7f_4a61_b347_75b27922fd39.slice/crio-617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc WatchSource:0}: Error finding container 617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc: Status 404 returned error can't find the container with id 617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc
Oct 31 03:18:35.432678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.432639    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f" event={"ID":"81fa838b-3e7f-4a61-b347-75b27922fd39","Type":"ContainerStarted","Data":"53400b32873c5a06c51ceb8d7ddd1d72232630d786cd8960e1194f96160ca5a5"}
Oct 31 03:18:35.433169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.432718    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f" event={"ID":"81fa838b-3e7f-4a61-b347-75b27922fd39","Type":"ContainerStarted","Data":"617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc"}
Oct 31 03:18:35.433169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.432911    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:35.433984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.433968    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:18:35.541361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:35.541256    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f" podStartSLOduration=1.5412402410000001 podStartE2EDuration="1.541240241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:35.538877519 +0000 UTC m=+665.553439227" watchObservedRunningTime="2026-10-31 03:18:35.541240241 +0000 UTC m=+665.555801949"
Oct 31 03:18:36.441365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:36.441320    5831 generic.go:356] "Generic (PLEG): container finished" podID="81fa838b-3e7f-4a61-b347-75b27922fd39" containerID="53400b32873c5a06c51ceb8d7ddd1d72232630d786cd8960e1194f96160ca5a5" exitCode=0
Oct 31 03:18:36.441819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:36.441425    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f" event={"ID":"81fa838b-3e7f-4a61-b347-75b27922fd39","Type":"ContainerDied","Data":"53400b32873c5a06c51ceb8d7ddd1d72232630d786cd8960e1194f96160ca5a5"}
Oct 31 03:18:36.695815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:36.695728    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:18:37.073150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.073125    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:18:37.153262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153234    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153296    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153316    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153346    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153373    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153391    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153435    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153521    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153562    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\") pod \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\" (UID: \"6f281655-8ff0-4542-98ae-8bb67c8bcb7d\") "
Oct 31 03:18:37.153846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153710    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:37.153846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:37.153846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153719    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:37.154000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.153946    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:37.154065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.154007    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.154065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.154032    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.154065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.154045    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.154283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.154200    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:37.156124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.156096    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:37.156321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.156300    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "kube-api-access-6jk7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:37.156510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.156491    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:37.156629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.156613    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo" pod "6f281655-8ff0-4542-98ae-8bb67c8bcb7d" (UID: "6f281655-8ff0-4542-98ae-8bb67c8bcb7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.254968    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.254999    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.255008    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.255017    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.255026    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jk7g\" (UniqueName: \"kubernetes.io/projected/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-kube-api-access-6jk7g\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.255053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.255035    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f281655-8ff0-4542-98ae-8bb67c8bcb7d-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.450744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.450711    5831 generic.go:356] "Generic (PLEG): container finished" podID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerID="e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726" exitCode=0
Oct 31 03:18:37.451255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.450784    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"
Oct 31 03:18:37.451255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.450801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerDied","Data":"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"}
Oct 31 03:18:37.451255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.450919    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f" event={"ID":"6f281655-8ff0-4542-98ae-8bb67c8bcb7d","Type":"ContainerDied","Data":"aa7497f4f42f81afe22d684ee65a50a2f691ca085705e361bae81dd270793c78"}
Oct 31 03:18:37.451255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.450948    5831 scope.go:122] "RemoveContainer" containerID="e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"
Oct 31 03:18:37.472094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.471209    5831 scope.go:122] "RemoveContainer" containerID="75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"
Oct 31 03:18:37.501707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.501675    5831 scope.go:122] "RemoveContainer" containerID="e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"
Oct 31 03:18:37.502693 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:37.502669    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726\": container with ID starting with e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726 not found: ID does not exist" containerID="e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"
Oct 31 03:18:37.502822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.502703    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726"} err="failed to get container status \"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726\": rpc error: code = NotFound desc = could not find container \"e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726\": container with ID starting with e0f1329e3735c67f586d827b41d14991ad0606dfb39e39c160cd0b399b3f3726 not found: ID does not exist"
Oct 31 03:18:37.502822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.502729    5831 scope.go:122] "RemoveContainer" containerID="75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"
Oct 31 03:18:37.503132 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:37.503097    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48\": container with ID starting with 75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48 not found: ID does not exist" containerID="75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"
Oct 31 03:18:37.503202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.503140    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48"} err="failed to get container status \"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48\": rpc error: code = NotFound desc = could not find container \"75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48\": container with ID starting with 75a6d12521e6b0a1954dad4ab7ee35bbc3ff2b8fffed5fa4ac9511603c971e48 not found: ID does not exist"
Oct 31 03:18:37.605835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.605805    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:18:37.609775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.609753    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4072c5d1-openshift-default-fcb679d7c-9ls8f"]
Oct 31 03:18:37.609995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.609980    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:37.669560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.660094    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\") pod \"81fa838b-3e7f-4a61-b347-75b27922fd39\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") "
Oct 31 03:18:37.669560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.660163    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\") pod \"81fa838b-3e7f-4a61-b347-75b27922fd39\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") "
Oct 31 03:18:37.669560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.660223    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\" (UniqueName: \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\") pod \"81fa838b-3e7f-4a61-b347-75b27922fd39\" (UID: \"81fa838b-3e7f-4a61-b347-75b27922fd39\") "
Oct 31 03:18:37.671095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.671056    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca" pod "81fa838b-3e7f-4a61-b347-75b27922fd39" (UID: "81fa838b-3e7f-4a61-b347-75b27922fd39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:37.673397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.673349    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca" pod "81fa838b-3e7f-4a61-b347-75b27922fd39" (UID: "81fa838b-3e7f-4a61-b347-75b27922fd39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:37.674717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.674652    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh" pod "81fa838b-3e7f-4a61-b347-75b27922fd39" (UID: "81fa838b-3e7f-4a61-b347-75b27922fd39"). InnerVolumeSpecName "kube-api-access-pkrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:37.761996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.761950    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.761996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.761998    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrlh\" (UniqueName: \"kubernetes.io/projected/81fa838b-3e7f-4a61-b347-75b27922fd39-kube-api-access-pkrlh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.761996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:37.762014    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fa838b-3e7f-4a61-b347-75b27922fd39-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.136622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.136594    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" path="/var/lib/kubelet/pods/6f281655-8ff0-4542-98ae-8bb67c8bcb7d/volumes"
Oct 31 03:18:38.349870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.349848    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:18:38.467390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.467350    5831 generic.go:356] "Generic (PLEG): container finished" podID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerID="af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0" exitCode=0
Oct 31 03:18:38.467894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.467414    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"
Oct 31 03:18:38.467894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.467439    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerDied","Data":"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"}
Oct 31 03:18:38.467894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.467557    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r" event={"ID":"f9c91dd7-390a-48df-918c-cc0c529462c1","Type":"ContainerDied","Data":"98a74fa72df0cfab22794d80b0ac36a018de66549589cc24c26496a43bae2dea"}
Oct 31 03:18:38.467894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.467571    5831 scope.go:122] "RemoveContainer" containerID="af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"
Oct 31 03:18:38.468899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.468877    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"
Oct 31 03:18:38.469114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.468898    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f" event={"ID":"81fa838b-3e7f-4a61-b347-75b27922fd39","Type":"ContainerDied","Data":"617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc"}
Oct 31 03:18:38.469114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.468945    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617412d2af7054de0140f4c4259c4d88b320e42de29f74fd8f15a665e8acffdc"
Oct 31 03:18:38.470674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470647    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.470817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470727    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.470882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470820    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.470937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470926    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470933    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:38.471000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470964    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.470998    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471044    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471090    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471117    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\") pod \"f9c91dd7-390a-48df-918c-cc0c529462c1\" (UID: \"f9c91dd7-390a-48df-918c-cc0c529462c1\") "
Oct 31 03:18:38.471566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471513    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:38.471673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471590    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.471974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.471933    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:38.472087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.472070    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:38.472219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.472197    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:38.474065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.474037    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:18:38.474240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.474216    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "kube-api-access-nhngc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:38.475016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.474912    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:38.475120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.475061    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token" pod "f9c91dd7-390a-48df-918c-cc0c529462c1" (UID: "f9c91dd7-390a-48df-918c-cc0c529462c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:38.482945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.482917    5831 scope.go:122] "RemoveContainer" containerID="f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"
Oct 31 03:18:38.498011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.497986    5831 scope.go:122] "RemoveContainer" containerID="af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"
Oct 31 03:18:38.498416 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:38.498385    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0\": container with ID starting with af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0 not found: ID does not exist" containerID="af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"
Oct 31 03:18:38.498519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.498428    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0"} err="failed to get container status \"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0\": rpc error: code = NotFound desc = could not find container \"af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0\": container with ID starting with af264e7840f781c4e7a25b55d01a2f2f9f70cd355cd1865d37902387d31eb6d0 not found: ID does not exist"
Oct 31 03:18:38.498519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.498455    5831 scope.go:122] "RemoveContainer" containerID="f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"
Oct 31 03:18:38.498836 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:38.498810    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f\": container with ID starting with f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f not found: ID does not exist" containerID="f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"
Oct 31 03:18:38.498902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.498846    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f"} err="failed to get container status \"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f\": rpc error: code = NotFound desc = could not find container \"f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f\": container with ID starting with f1276182e17548cdce92750e932a3f870d9b33dd2e2afde6c48b7b899020281f not found: ID does not exist"
Oct 31 03:18:38.572712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572683    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572709    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572719    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572728    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhngc\" (UniqueName: \"kubernetes.io/projected/f9c91dd7-390a-48df-918c-cc0c529462c1-kube-api-access-nhngc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572739    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572747    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572754    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c91dd7-390a-48df-918c-cc0c529462c1-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.572921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.572762    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c91dd7-390a-48df-918c-cc0c529462c1-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.880624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.880590    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:18:38.885071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:38.885044    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bac33a89-gw-openshift-default-68df85649-2m84r"]
Oct 31 03:18:39.245993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.245953    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:18:39.309546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.309502    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:18:39.321116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.321071    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" podUID="5761a401-6791-494c-a37d-eb1d22c94ece" containerName="payload-processing" containerID="cri-o://6122e49c31b66281ff9b88a53c19c0949cdd8e2c57eb0657532aa0ff6b767e95" gracePeriod=25
Oct 31 03:18:39.419281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.419227    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" podUID="069ee20a-62fe-408b-97ed-9392074a2c3d" containerName="payload-pre-processing" containerID="cri-o://e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6" gracePeriod=25
Oct 31 03:18:39.480246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.480206    5831 generic.go:356] "Generic (PLEG): container finished" podID="5761a401-6791-494c-a37d-eb1d22c94ece" containerID="6122e49c31b66281ff9b88a53c19c0949cdd8e2c57eb0657532aa0ff6b767e95" exitCode=0
Oct 31 03:18:39.480655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.480298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" event={"ID":"5761a401-6791-494c-a37d-eb1d22c94ece","Type":"ContainerDied","Data":"6122e49c31b66281ff9b88a53c19c0949cdd8e2c57eb0657532aa0ff6b767e95"}
Oct 31 03:18:39.589057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.589023    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:18:39.678314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.678286    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:18:39.685199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.685173    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\") pod \"5761a401-6791-494c-a37d-eb1d22c94ece\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") "
Oct 31 03:18:39.685333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.685225    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\" (UniqueName: \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\") pod \"5761a401-6791-494c-a37d-eb1d22c94ece\" (UID: \"5761a401-6791-494c-a37d-eb1d22c94ece\") "
Oct 31 03:18:39.685653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.685620    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume" pod "5761a401-6791-494c-a37d-eb1d22c94ece" (UID: "5761a401-6791-494c-a37d-eb1d22c94ece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:39.687470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.687452    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp" pod "5761a401-6791-494c-a37d-eb1d22c94ece" (UID: "5761a401-6791-494c-a37d-eb1d22c94ece"). InnerVolumeSpecName "kube-api-access-fvgrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:39.786226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.786140    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\") pod \"069ee20a-62fe-408b-97ed-9392074a2c3d\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") "
Oct 31 03:18:39.786226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.786177    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\" (UniqueName: \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\") pod \"069ee20a-62fe-408b-97ed-9392074a2c3d\" (UID: \"069ee20a-62fe-408b-97ed-9392074a2c3d\") "
Oct 31 03:18:39.786406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.786396    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5761a401-6791-494c-a37d-eb1d22c94ece-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.786475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.786410    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgrp\" (UniqueName: \"kubernetes.io/projected/5761a401-6791-494c-a37d-eb1d22c94ece-kube-api-access-fvgrp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.786584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.786558    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume" pod "069ee20a-62fe-408b-97ed-9392074a2c3d" (UID: "069ee20a-62fe-408b-97ed-9392074a2c3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:39.788412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.788394    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg" pod "069ee20a-62fe-408b-97ed-9392074a2c3d" (UID: "069ee20a-62fe-408b-97ed-9392074a2c3d"). InnerVolumeSpecName "kube-api-access-d9mtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:39.834140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.834100    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"]
Oct 31 03:18:39.835294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.835267    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f427162-k2lg8"]
Oct 31 03:18:39.886994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.886960    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069ee20a-62fe-408b-97ed-9392074a2c3d-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.886994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:39.886990    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mtg\" (UniqueName: \"kubernetes.io/projected/069ee20a-62fe-408b-97ed-9392074a2c3d-kube-api-access-d9mtg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:40.136114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.136038    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a5c52e-7ce5-41bc-bd06-364dbd296fba" path="/var/lib/kubelet/pods/c8a5c52e-7ce5-41bc-bd06-364dbd296fba/volumes"
Oct 31 03:18:40.136561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.136545    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" path="/var/lib/kubelet/pods/f9c91dd7-390a-48df-918c-cc0c529462c1/volumes"
Oct 31 03:18:40.487820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.487785    5831 generic.go:356] "Generic (PLEG): container finished" podID="069ee20a-62fe-408b-97ed-9392074a2c3d" containerID="e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6" exitCode=0
Oct 31 03:18:40.488327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.487840    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"
Oct 31 03:18:40.488327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.487885    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" event={"ID":"069ee20a-62fe-408b-97ed-9392074a2c3d","Type":"ContainerDied","Data":"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"}
Oct 31 03:18:40.488327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.487943    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph" event={"ID":"069ee20a-62fe-408b-97ed-9392074a2c3d","Type":"ContainerDied","Data":"1c2e37a789ecf9167d2d14f5c4b3d8f1dfc4c2fb0fe15c45e267c457db5f2e50"}
Oct 31 03:18:40.488327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.487977    5831 scope.go:122] "RemoveContainer" containerID="e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"
Oct 31 03:18:40.489461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.489244    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt" event={"ID":"5761a401-6791-494c-a37d-eb1d22c94ece","Type":"ContainerDied","Data":"faa3396b7f6bb5c016ed2c2d4f51c488ae9f68bcd249ffc3294e0082423548aa"}
Oct 31 03:18:40.489461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.489250    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"
Oct 31 03:18:40.499211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.499181    5831 scope.go:122] "RemoveContainer" containerID="e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"
Oct 31 03:18:40.499438 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:40.499413    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6\": container with ID starting with e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6 not found: ID does not exist" containerID="e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"
Oct 31 03:18:40.499500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.499445    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6"} err="failed to get container status \"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6\": rpc error: code = NotFound desc = could not find container \"e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6\": container with ID starting with e204ec2e9aecb64831c5d4c21933e14d8df0351a7a3847728c6799861b802cf6 not found: ID does not exist"
Oct 31 03:18:40.499500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.499463    5831 scope.go:122] "RemoveContainer" containerID="6122e49c31b66281ff9b88a53c19c0949cdd8e2c57eb0657532aa0ff6b767e95"
Oct 31 03:18:40.640775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.640740    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:18:40.686800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.686763    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f427162-dfd89866c-qxfph"]
Oct 31 03:18:40.745158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.745074    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:18:40.748764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:40.748735    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f427162-574d49d87-nbrpt"]
Oct 31 03:18:42.133807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.133776    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069ee20a-62fe-408b-97ed-9392074a2c3d" path="/var/lib/kubelet/pods/069ee20a-62fe-408b-97ed-9392074a2c3d/volumes"
Oct 31 03:18:42.134183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.134139    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5761a401-6791-494c-a37d-eb1d22c94ece" path="/var/lib/kubelet/pods/5761a401-6791-494c-a37d-eb1d22c94ece/volumes"
Oct 31 03:18:42.518653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.518625    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:18:42.519049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519037    5831 cpu_manager.go:414] "containerMap: removing container" podUID="069ee20a-62fe-408b-97ed-9392074a2c3d" containerName="payload-pre-processing"
Oct 31 03:18:42.519097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519051    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069ee20a-62fe-408b-97ed-9392074a2c3d" containerName="payload-pre-processing"
Oct 31 03:18:42.519097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519077    5831 cpu_manager.go:414] "containerMap: removing container" podUID="81fa838b-3e7f-4a61-b347-75b27922fd39" containerName="revoke-keys"
Oct 31 03:18:42.519097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519082    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fa838b-3e7f-4a61-b347-75b27922fd39" containerName="revoke-keys"
Oct 31 03:18:42.519097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519094    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5761a401-6791-494c-a37d-eb1d22c94ece" containerName="payload-processing"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519100    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5761a401-6791-494c-a37d-eb1d22c94ece" containerName="payload-processing"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519112    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519117    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519123    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519127    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519137    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519142    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519206    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:42.519216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519213    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:42.519455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519221    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="069ee20a-62fe-408b-97ed-9392074a2c3d" containerName="payload-pre-processing"
Oct 31 03:18:42.519455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519229    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fa838b-3e7f-4a61-b347-75b27922fd39" containerName="revoke-keys"
Oct 31 03:18:42.519455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.519235    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5761a401-6791-494c-a37d-eb1d22c94ece" containerName="payload-processing"
Oct 31 03:18:42.524277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.524253    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.589012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.588980    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9360eb-openshift-default-dockercfg-lsbz7\""
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.613649    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.613737    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.613893    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.613957    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.613987    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jsl\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.614030    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.614061    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.614093    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.614451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.614145    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.636329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.636287    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:18:42.715601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715560    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715619    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715679    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715809    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715845    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jsl\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715880    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.715918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715907    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715941    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.715994    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.716122    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.716379    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.716405    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.716387    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.716553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.716507    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.718329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.718308    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.718561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.718520    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.769022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.768947    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.769224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.769200    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jsl\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.838443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:42.838403    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:42.978433 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:18:42.978392    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:18:43.265293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.265271    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:18:43.267883 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:43.267852    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc686c0cc_a050_45dc_be8f_e934579c4db0.slice/crio-792764ee9a73eb5afbae06af31f799a74619450095ddff900fd8becbb31ef39e WatchSource:0}: Error finding container 792764ee9a73eb5afbae06af31f799a74619450095ddff900fd8becbb31ef39e: Status 404 returned error can't find the container with id 792764ee9a73eb5afbae06af31f799a74619450095ddff900fd8becbb31ef39e
Oct 31 03:18:43.270289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.270257    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:43.270373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.270324    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:43.270373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.270356    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:18:43.520355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.520230    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerStarted","Data":"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"}
Oct 31 03:18:43.520510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.520388    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerStarted","Data":"792764ee9a73eb5afbae06af31f799a74619450095ddff900fd8becbb31ef39e"}
Oct 31 03:18:43.633063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.633009    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podStartSLOduration=1.632990848 podStartE2EDuration="1.632990848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:43.630657035 +0000 UTC m=+673.645218743" watchObservedRunningTime="2026-10-31 03:18:43.632990848 +0000 UTC m=+673.647552556"
Oct 31 03:18:43.839358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:43.839274    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:44.247652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.247613    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" podUID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" containerName="payload-processing" containerID="cri-o://0da3e404e1bc1ea25b8060d2581fe395f223043edea40aae7482d923dae07f9b" gracePeriod=25
Oct 31 03:18:44.309790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.309752    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" podUID="8bbccfca-df17-4416-b174-a634e81b0254" containerName="payload-pre-processing" containerID="cri-o://d4f04ddf84a884c54ce04cadf11aaa539951ffcb6d7374dccbba6b56d5c20234" gracePeriod=25
Oct 31 03:18:44.532602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.532568    5831 generic.go:356] "Generic (PLEG): container finished" podID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" containerID="0da3e404e1bc1ea25b8060d2581fe395f223043edea40aae7482d923dae07f9b" exitCode=0
Oct 31 03:18:44.532775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.532565    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" event={"ID":"4509f7c8-881e-4deb-b9e7-3ded6e6930d8","Type":"ContainerDied","Data":"0da3e404e1bc1ea25b8060d2581fe395f223043edea40aae7482d923dae07f9b"}
Oct 31 03:18:44.535592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.535558    5831 generic.go:356] "Generic (PLEG): container finished" podID="8bbccfca-df17-4416-b174-a634e81b0254" containerID="d4f04ddf84a884c54ce04cadf11aaa539951ffcb6d7374dccbba6b56d5c20234" exitCode=0
Oct 31 03:18:44.535757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.535565    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" event={"ID":"8bbccfca-df17-4416-b174-a634e81b0254","Type":"ContainerDied","Data":"d4f04ddf84a884c54ce04cadf11aaa539951ffcb6d7374dccbba6b56d5c20234"}
Oct 31 03:18:44.634598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.634341    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:18:44.640479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.640455    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:18:44.737245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737216    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\") pod \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") "
Oct 31 03:18:44.737245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737254    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\" (UniqueName: \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\") pod \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\" (UID: \"4509f7c8-881e-4deb-b9e7-3ded6e6930d8\") "
Oct 31 03:18:44.737505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737352    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\" (UniqueName: \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\") pod \"8bbccfca-df17-4416-b174-a634e81b0254\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") "
Oct 31 03:18:44.737505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737414    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\") pod \"8bbccfca-df17-4416-b174-a634e81b0254\" (UID: \"8bbccfca-df17-4416-b174-a634e81b0254\") "
Oct 31 03:18:44.737800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737772    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume" pod "4509f7c8-881e-4deb-b9e7-3ded6e6930d8" (UID: "4509f7c8-881e-4deb-b9e7-3ded6e6930d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:44.737933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.737801    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume" pod "8bbccfca-df17-4416-b174-a634e81b0254" (UID: "8bbccfca-df17-4416-b174-a634e81b0254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:44.739643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.739618    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc" pod "4509f7c8-881e-4deb-b9e7-3ded6e6930d8" (UID: "4509f7c8-881e-4deb-b9e7-3ded6e6930d8"). InnerVolumeSpecName "kube-api-access-5wcvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:44.740937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.740912    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp" pod "8bbccfca-df17-4416-b174-a634e81b0254" (UID: "8bbccfca-df17-4416-b174-a634e81b0254"). InnerVolumeSpecName "kube-api-access-z8nhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:44.838812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.838659    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"]
Oct 31 03:18:44.842770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.841368    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:44.842770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.841402    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wcvc\" (UniqueName: \"kubernetes.io/projected/4509f7c8-881e-4deb-b9e7-3ded6e6930d8-kube-api-access-5wcvc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:44.842770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.841420    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8nhp\" (UniqueName: \"kubernetes.io/projected/8bbccfca-df17-4416-b174-a634e81b0254-kube-api-access-z8nhp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:44.842770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.841435    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbccfca-df17-4416-b174-a634e81b0254-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:44.847986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.847953    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15000fe0-gsvz9"]
Oct 31 03:18:44.855559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:44.851880    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:45.161294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161213    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 03:18:45.161682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161667    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" containerName="payload-processing"
Oct 31 03:18:45.161737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161683    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" containerName="payload-processing"
Oct 31 03:18:45.161737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161708    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8bbccfca-df17-4416-b174-a634e81b0254" containerName="payload-pre-processing"
Oct 31 03:18:45.161737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161713    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbccfca-df17-4416-b174-a634e81b0254" containerName="payload-pre-processing"
Oct 31 03:18:45.161737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161731    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:45.161737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161736    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:45.161927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161796    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f281655-8ff0-4542-98ae-8bb67c8bcb7d" containerName="istio-proxy"
Oct 31 03:18:45.161927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161803    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c91dd7-390a-48df-918c-cc0c529462c1" containerName="istio-proxy"
Oct 31 03:18:45.161927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161809    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" containerName="payload-processing"
Oct 31 03:18:45.161927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.161817    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbccfca-df17-4416-b174-a634e81b0254" containerName="payload-pre-processing"
Oct 31 03:18:45.165239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.165219    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.216051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.216019    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b59563\""
Oct 31 03:18:45.219071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.219044    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 03:18:45.263783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.248722    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq45f\" (UniqueName: \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.263783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.248802    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.294944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.294869    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 03:18:45.300826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.300800    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.312748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.312553    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b59563-dockercfg-6h7f5\""
Oct 31 03:18:45.322582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.322510    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 03:18:45.349832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.349796    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq45f\" (UniqueName: \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.350028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.349868    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.350648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.350629    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.403073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.403038    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq45f\" (UniqueName: \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\") pod \"payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.451268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.451237    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.451453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.451293    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbg78\" (UniqueName: \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.479069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.479044    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.558274    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.558330    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbg78\" (UniqueName: \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.558639    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf" event={"ID":"4509f7c8-881e-4deb-b9e7-3ded6e6930d8","Type":"ContainerDied","Data":"f430a8a0be9afb5c0d652f99c3b0bcced8d0f2679e17a767408bf69e7e7f0fe2"}
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.558705    5831 scope.go:122] "RemoveContainer" containerID="0da3e404e1bc1ea25b8060d2581fe395f223043edea40aae7482d923dae07f9b"
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.558921    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"
Oct 31 03:18:45.561062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.559436    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.562278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.562248    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl" event={"ID":"8bbccfca-df17-4416-b174-a634e81b0254","Type":"ContainerDied","Data":"ead635dd8042e71ec11f803abc095a21b379085cad1691d44b0da3debb13486c"}
Oct 31 03:18:45.562865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.562664    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:45.562865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.562753    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"
Oct 31 03:18:45.563715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.563695    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:18:45.578633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.578438    5831 scope.go:122] "RemoveContainer" containerID="d4f04ddf84a884c54ce04cadf11aaa539951ffcb6d7374dccbba6b56d5c20234"
Oct 31 03:18:45.611955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.611924    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbg78\" (UniqueName: \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\") pod \"payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.617826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.617800    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:45.712464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.712372    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:18:45.717749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.717719    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15000fe0-6d64fb97f5-5c4sl"]
Oct 31 03:18:45.847809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.847779    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 03:18:45.854702 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:45.854634    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c78470_f788_48aa_9b33_f6f23a76825a.slice/crio-307cd102638e46bc3e5b35592c17dc4d889f41d1c55187500bba354c4abae44f WatchSource:0}: Error finding container 307cd102638e46bc3e5b35592c17dc4d889f41d1c55187500bba354c4abae44f: Status 404 returned error can't find the container with id 307cd102638e46bc3e5b35592c17dc4d889f41d1c55187500bba354c4abae44f
Oct 31 03:18:45.929594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.929566    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:18:45.933932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.933906    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15000fe0-9676fd9cc-l85jf"]
Oct 31 03:18:45.996360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:45.996284    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 03:18:45.998928 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:45.998896    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d35f37a_64cf_492c_82d2_935cf283aa56.slice/crio-58bf34f22c5dfd0de5578b1ecdd7e02b92c3e5a75c90192f03526754b269a895 WatchSource:0}: Error finding container 58bf34f22c5dfd0de5578b1ecdd7e02b92c3e5a75c90192f03526754b269a895: Status 404 returned error can't find the container with id 58bf34f22c5dfd0de5578b1ecdd7e02b92c3e5a75c90192f03526754b269a895
Oct 31 03:18:46.135079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.135047    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4509f7c8-881e-4deb-b9e7-3ded6e6930d8" path="/var/lib/kubelet/pods/4509f7c8-881e-4deb-b9e7-3ded6e6930d8/volumes"
Oct 31 03:18:46.135418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.135405    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbccfca-df17-4416-b174-a634e81b0254" path="/var/lib/kubelet/pods/8bbccfca-df17-4416-b174-a634e81b0254/volumes"
Oct 31 03:18:46.135805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.135791    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c545aa-d97b-4fbb-8ff6-270ef46d7078" path="/var/lib/kubelet/pods/f7c545aa-d97b-4fbb-8ff6-270ef46d7078/volumes"
Oct 31 03:18:46.609962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.608959    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" event={"ID":"f4c78470-f788-48aa-9b33-f6f23a76825a","Type":"ContainerStarted","Data":"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"}
Oct 31 03:18:46.609962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.609030    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" event={"ID":"f4c78470-f788-48aa-9b33-f6f23a76825a","Type":"ContainerStarted","Data":"307cd102638e46bc3e5b35592c17dc4d889f41d1c55187500bba354c4abae44f"}
Oct 31 03:18:46.609962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.609898    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:46.613224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.612784    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" event={"ID":"1d35f37a-64cf-492c-82d2-935cf283aa56","Type":"ContainerStarted","Data":"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"}
Oct 31 03:18:46.613224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.612833    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" event={"ID":"1d35f37a-64cf-492c-82d2-935cf283aa56","Type":"ContainerStarted","Data":"58bf34f22c5dfd0de5578b1ecdd7e02b92c3e5a75c90192f03526754b269a895"}
Oct 31 03:18:46.613224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.612887    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:46.847176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.847108    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" podStartSLOduration=1.847087015 podStartE2EDuration="1.847087015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:46.739019493 +0000 UTC m=+676.753581227" watchObservedRunningTime="2026-10-31 03:18:46.847087015 +0000 UTC m=+676.861648723"
Oct 31 03:18:46.850263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:46.849168    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" podStartSLOduration=1.849151482 podStartE2EDuration="1.849151482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:46.84697401 +0000 UTC m=+676.861535718" watchObservedRunningTime="2026-10-31 03:18:46.849151482 +0000 UTC m=+676.863713194"
Oct 31 03:18:47.900277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:47.900231    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 03:18:47.905701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:47.905674    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:47.984991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:47.984953    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:47.985177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:47.985035    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstwd\" (UniqueName: \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.003395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.003357    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-dd5b85-dockercfg-gnwln\""
Oct 31 03:18:48.003395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.003366    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-dd5b85\""
Oct 31 03:18:48.007229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.007203    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 03:18:48.086371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.086338    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.086556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.086395    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstwd\" (UniqueName: \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.087295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.087272    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.140833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.140802    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstwd\" (UniqueName: \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\") pod \"payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.221278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.221237    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:48.599505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.599482    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 03:18:48.601811 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:48.601780    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06384da_1a15_4600_a7df_74c26605e7ec.slice/crio-cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1 WatchSource:0}: Error finding container cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1: Status 404 returned error can't find the container with id cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1
Oct 31 03:18:48.646472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:48.646434    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" event={"ID":"a06384da-1a15-4600-a7df-74c26605e7ec","Type":"ContainerStarted","Data":"cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1"}
Oct 31 03:18:49.655062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:49.655025    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" event={"ID":"a06384da-1a15-4600-a7df-74c26605e7ec","Type":"ContainerStarted","Data":"85ae2038721d4117350afb4577d0886aba0466ba0ede3b2a8eec020e971b1656"}
Oct 31 03:18:49.655474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:49.655185    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:18:53.965389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:53.965328    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" podStartSLOduration=6.965313863 podStartE2EDuration="6.965313863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:49.823177277 +0000 UTC m=+679.837738984" watchObservedRunningTime="2026-10-31 03:18:53.965313863 +0000 UTC m=+683.979875571"
Oct 31 03:18:53.965877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:53.965667    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:18:57.612758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.612720    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 03:18:57.617781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.617760    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.634345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.634321    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 03:18:57.709153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.709109    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtd6\" (UniqueName: \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.722807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.722775    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.723840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.716075    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9360eb-dockercfg-l52vx\""
Oct 31 03:18:57.724018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.715487    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9360eb\""
Oct 31 03:18:57.726306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.726281    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 03:18:57.764400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.764372    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 03:18:57.768241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.768224    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:57.818121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.818085    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 03:18:57.824908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.824877    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtd6\" (UniqueName: \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.825086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.825014    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.825981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.825956    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.878612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.878514    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtd6\" (UniqueName: \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\") pod \"payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:57.926221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.926187    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:57.926395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.926375    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhvf\" (UniqueName: \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:57.933022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:57.932997    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:58.028074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.028035    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhvf\" (UniqueName: \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:58.028235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.028119    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:58.028901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.028880    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:58.083503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.083472    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhvf\" (UniqueName: \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\") pod \"payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:58.314337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.314158    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 03:18:58.317248 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:58.317219    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb03491_33e0_4f12_b262_7042c4e58958.slice/crio-7aec27a841c3106e069b0a954d85cc456f6051fd156c89cce92bf2a286cde4cd WatchSource:0}: Error finding container 7aec27a841c3106e069b0a954d85cc456f6051fd156c89cce92bf2a286cde4cd: Status 404 returned error can't find the container with id 7aec27a841c3106e069b0a954d85cc456f6051fd156c89cce92bf2a286cde4cd
Oct 31 03:18:58.383625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.383603    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:58.649576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.649488    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 03:18:58.729735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.729699    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" event={"ID":"5eb03491-33e0-4f12-b262-7042c4e58958","Type":"ContainerStarted","Data":"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"}
Oct 31 03:18:58.729928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.729763    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" event={"ID":"5eb03491-33e0-4f12-b262-7042c4e58958","Type":"ContainerStarted","Data":"7aec27a841c3106e069b0a954d85cc456f6051fd156c89cce92bf2a286cde4cd"}
Oct 31 03:18:58.729928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.729793    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:18:58.752940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.752918    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 03:18:58.755050 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:18:58.755022    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3f1f8f_18ae_452a_9554_3bc17d7c5551.slice/crio-2585d25457f41346dc77cb9c64b13fd24476af5eb18563febef7905b82a4458c WatchSource:0}: Error finding container 2585d25457f41346dc77cb9c64b13fd24476af5eb18563febef7905b82a4458c: Status 404 returned error can't find the container with id 2585d25457f41346dc77cb9c64b13fd24476af5eb18563febef7905b82a4458c
Oct 31 03:18:58.861235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:58.861185    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" podStartSLOduration=1.861172135 podStartE2EDuration="1.861172135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:58.860972871 +0000 UTC m=+688.875534580" watchObservedRunningTime="2026-10-31 03:18:58.861172135 +0000 UTC m=+688.875733839"
Oct 31 03:18:59.738696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:59.738653    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" event={"ID":"cb3f1f8f-18ae-452a-9554-3bc17d7c5551","Type":"ContainerStarted","Data":"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"}
Oct 31 03:18:59.739163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:59.738716    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" event={"ID":"cb3f1f8f-18ae-452a-9554-3bc17d7c5551","Type":"ContainerStarted","Data":"2585d25457f41346dc77cb9c64b13fd24476af5eb18563febef7905b82a4458c"}
Oct 31 03:18:59.739163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:59.738972    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:18:59.844662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:18:59.844596    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" podStartSLOduration=2.8445816600000002 podStartE2EDuration="2.84458166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.844002287 +0000 UTC m=+689.858564119" watchObservedRunningTime="2026-10-31 03:18:59.84458166 +0000 UTC m=+689.859143369"
Oct 31 03:19:00.665214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:00.665175    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 03:19:02.599558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.594096    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 03:19:02.613425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.613391    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.647260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.647230    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 03:19:02.700556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.693871    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d39257\""
Oct 31 03:19:02.700556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.694054    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d39257-dockercfg-9ljqd\""
Oct 31 03:19:02.738835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.737266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsvc\" (UniqueName: \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.738835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.737369    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.743959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.743926    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 03:19:02.747920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.747894    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:02.810938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.810904    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 03:19:02.838159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.838126    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsvc\" (UniqueName: \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.838365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.838187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.838365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.838218    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:02.838365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.838290    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6mr\" (UniqueName: \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:02.839162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.839134    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.933824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.933778    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsvc\" (UniqueName: \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\") pod \"payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:02.939110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.939055    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6mr\" (UniqueName: \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:02.939274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.939194    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:02.940039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:02.940014    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:03.006402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.006369    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6mr\" (UniqueName: \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\") pod \"payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:03.070724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.070688    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:03.229435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.229284    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:03.467907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.467877    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 03:19:03.470359 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:19:03.470331    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d16824e_2576_4aa9_9d2b_8ff73343732c.slice/crio-142cf95c66765f33a92bd28e5be5c39897feaf401ebd2b48cf5856340f90ecc5 WatchSource:0}: Error finding container 142cf95c66765f33a92bd28e5be5c39897feaf401ebd2b48cf5856340f90ecc5: Status 404 returned error can't find the container with id 142cf95c66765f33a92bd28e5be5c39897feaf401ebd2b48cf5856340f90ecc5
Oct 31 03:19:03.634789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.634657    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 03:19:03.637721 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:19:03.637693    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92577884_e162_433f_ab59_50b7ff56a1af.slice/crio-7cdd0916a6af0102f8aafd61980836953b214b92c3ad14551725f0116033d029 WatchSource:0}: Error finding container 7cdd0916a6af0102f8aafd61980836953b214b92c3ad14551725f0116033d029: Status 404 returned error can't find the container with id 7cdd0916a6af0102f8aafd61980836953b214b92c3ad14551725f0116033d029
Oct 31 03:19:03.778478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.778378    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" event={"ID":"8d16824e-2576-4aa9-9d2b-8ff73343732c","Type":"ContainerStarted","Data":"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"}
Oct 31 03:19:03.778675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.778554    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:03.778675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.778587    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" event={"ID":"8d16824e-2576-4aa9-9d2b-8ff73343732c","Type":"ContainerStarted","Data":"142cf95c66765f33a92bd28e5be5c39897feaf401ebd2b48cf5856340f90ecc5"}
Oct 31 03:19:03.805555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.805463    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" event={"ID":"92577884-e162-433f-ab59-50b7ff56a1af","Type":"ContainerStarted","Data":"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"}
Oct 31 03:19:03.805555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.805554    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" event={"ID":"92577884-e162-433f-ab59-50b7ff56a1af","Type":"ContainerStarted","Data":"7cdd0916a6af0102f8aafd61980836953b214b92c3ad14551725f0116033d029"}
Oct 31 03:19:03.806555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.806204    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:03.905006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:03.904940    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" podStartSLOduration=1.904917929 podStartE2EDuration="1.904917929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:03.903733707 +0000 UTC m=+693.918295419" watchObservedRunningTime="2026-10-31 03:19:03.904917929 +0000 UTC m=+693.919479638"
Oct 31 03:19:04.012142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:04.012073    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" podStartSLOduration=2.012056022 podStartE2EDuration="2.012056022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:04.010664977 +0000 UTC m=+694.025226684" watchObservedRunningTime="2026-10-31 03:19:04.012056022 +0000 UTC m=+694.026617731"
Oct 31 03:19:04.219764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:04.219729    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"]
Oct 31 03:19:04.223335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:04.223308    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46d24f95-wlq2f"]
Oct 31 03:19:04.719567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:04.716058    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:19:04.719567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:04.716827    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" containerID="cri-o://09ab551ac88d3bf73b828e6d76a219a00dba3f0f4a7667c31c5f06d098288470" gracePeriod=30
Oct 31 03:19:06.163715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:06.163681    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fa838b-3e7f-4a61-b347-75b27922fd39" path="/var/lib/kubelet/pods/81fa838b-3e7f-4a61-b347-75b27922fd39/volumes"
Oct 31 03:19:07.520628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:07.520314    5831 patch_prober.go:29] interesting pod/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 03:19:07.520628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:07.520381    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:19:09.742242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:09.742217    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 03:19:09.859853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:09.859816    5831 generic.go:356] "Generic (PLEG): container finished" podID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerID="09ab551ac88d3bf73b828e6d76a219a00dba3f0f4a7667c31c5f06d098288470" exitCode=0
Oct 31 03:19:09.860016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:09.859994    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerDied","Data":"09ab551ac88d3bf73b828e6d76a219a00dba3f0f4a7667c31c5f06d098288470"}
Oct 31 03:19:09.860078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:09.860062    5831 scope.go:122] "RemoveContainer" containerID="71e5826b0152cd9191e9c1a474f72ae35adc1b049e37b53e23677c3940cbd771"
Oct 31 03:19:10.010155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.010128    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:19:10.133802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133773    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133815    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133865    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133902    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133940    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.133973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.134014    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.134054    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.134090    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\") pod \"67f5156a-c641-4ec4-aa19-3a504880b23c\" (UID: \"67f5156a-c641-4ec4-aa19-3a504880b23c\") "
Oct 31 03:19:10.134725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.134701    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:19:10.135791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.135374    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:19:10.135791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.135398    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:19:10.135791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.135622    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:19:10.136172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.136139    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:19:10.138252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.138221    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:19:10.138382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.138357    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:19:10.138382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.138253    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:19:10.138840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.138792    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7" pod "67f5156a-c641-4ec4-aa19-3a504880b23c" (UID: "67f5156a-c641-4ec4-aa19-3a504880b23c"). InnerVolumeSpecName "kube-api-access-lf9r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:19:10.235216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235181    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235211    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235221    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235230    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235238    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f5156a-c641-4ec4-aa19-3a504880b23c-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235246    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235255    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf9r7\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-kube-api-access-lf9r7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235264    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f5156a-c641-4ec4-aa19-3a504880b23c-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.235453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.235273    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f5156a-c641-4ec4-aa19-3a504880b23c-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:19:10.741751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.741709    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:19:10.753508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.753484    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 03:19:10.868048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.868008    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8" event={"ID":"67f5156a-c641-4ec4-aa19-3a504880b23c","Type":"ContainerDied","Data":"4f48877e61d59ddeb35ed172d41b037661048ac397b484f868344a3826d13c95"}
Oct 31 03:19:10.868048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.868035    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"
Oct 31 03:19:10.870409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.868081    5831 scope.go:122] "RemoveContainer" containerID="09ab551ac88d3bf73b828e6d76a219a00dba3f0f4a7667c31c5f06d098288470"
Oct 31 03:19:10.984046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:10.984007    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:19:11.037561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.034667    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-46d24f95-openshift-default-66b9cf9465-tdss8"]
Oct 31 03:19:11.877376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.877344    5831 generic.go:356] "Generic (PLEG): container finished" podID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerID="dfad3a5cce56281a765506388af29575945554fd956b6f036236f01090f01563" exitCode=0
Oct 31 03:19:11.877816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.877398    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerDied","Data":"dfad3a5cce56281a765506388af29575945554fd956b6f036236f01090f01563"}
Oct 31 03:19:11.877881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.877863    5831 scope.go:122] "RemoveContainer" containerID="dfad3a5cce56281a765506388af29575945554fd956b6f036236f01090f01563"
Oct 31 03:19:11.890564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.879214    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:11.890564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.885648    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:11.890564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:11.885713    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:12.134956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:12.134880    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" path="/var/lib/kubelet/pods/67f5156a-c641-4ec4-aa19-3a504880b23c/volumes"
Oct 31 03:19:12.899100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:12.899055    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerStarted","Data":"35c058db943e2804d71615c8c5f2e6060e02060ccc7b1bf1e8f87c9e98bd2dfd"}
Oct 31 03:19:12.982555 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:19:12.978642    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:19:13.474006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:13.473975    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:19:13.474286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:13.474131    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:19:13.474393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:13.474306    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 03:19:14.474155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.474110    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:19:14.474671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.474181    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 03:19:14.698577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.696167    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 03:19:14.715067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.715029    5831 cpu_manager.go:414] "containerMap: removing container" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.715330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.715310    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.715477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.715465    5831 cpu_manager.go:414] "containerMap: removing container" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.715604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.715590    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.715873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.715856    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.721089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.721068    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:14.750546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.750499    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 03:19:14.785557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.784356    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:14.785557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.784456    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v542k\" (UniqueName: \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:14.820486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.802835    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7778d0-dockercfg-pnhjf\""
Oct 31 03:19:14.820486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.802954    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7778d0\""
Oct 31 03:19:14.834749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.834712    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 03:19:14.861055 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.861027    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 03:19:14.862170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.862143    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f5156a-c641-4ec4-aa19-3a504880b23c" containerName="istio-proxy"
Oct 31 03:19:14.866457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.866434    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:14.914039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.896912    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw5p\" (UniqueName: \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:14.926977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.919863    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:14.946766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.946730    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:14.947054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.947028    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v542k\" (UniqueName: \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:14.947191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.947168    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 03:19:14.948225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:14.947706    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:15.019605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.017233    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v542k\" (UniqueName: \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\") pod \"payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:15.042957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.042924    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:15.048767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.048731    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw5p\" (UniqueName: \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:15.048915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.048808    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:15.049754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.049727    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:15.165973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.165484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw5p\" (UniqueName: \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\") pod \"payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:15.191555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.188781    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:15.473646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.473605    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:19:15.473833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.473678    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 03:19:15.578550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.578450    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 03:19:15.737456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.737428    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 03:19:15.834611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.833931    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 03:19:15.953656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.953606    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" event={"ID":"8670eaf6-7e04-4dd1-9692-ca0795078bd2","Type":"ContainerStarted","Data":"dacc5e7e6ab084f0fcd10cb102e656c8ac56e3ccfda5e0f80093f6deeb1d94f7"}
Oct 31 03:19:15.953926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.953675    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" event={"ID":"8670eaf6-7e04-4dd1-9692-ca0795078bd2","Type":"ContainerStarted","Data":"dfdbdc27f801e6b7cfbac5a46b01050a53fc64565b28ef6376b8144b144cabe9"}
Oct 31 03:19:15.953926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.953709    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:15.955275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.955251    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" event={"ID":"5e471678-5fc3-4ab8-85b5-cd34b0310337","Type":"ContainerStarted","Data":"5fe2ee7a3d955d6201a815ac83ed1f67a02e1055ae22b676fda8608ec051d07d"}
Oct 31 03:19:15.955382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.955298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" event={"ID":"5e471678-5fc3-4ab8-85b5-cd34b0310337","Type":"ContainerStarted","Data":"5b96f7caad0f51cfa4c0f7b7218300936f403b6dc861fd126840d67dbeaa2ef7"}
Oct 31 03:19:15.955382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:15.955327    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:16.102759 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:16.096593    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" podStartSLOduration=2.096573686 podStartE2EDuration="2.096573686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:16.094935961 +0000 UTC m=+706.109497670" watchObservedRunningTime="2026-10-31 03:19:16.096573686 +0000 UTC m=+706.111135394"
Oct 31 03:19:16.477286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:16.477250    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:19:16.477594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:16.477567    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 03:19:17.474275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:17.474241    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:19:17.474674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:17.474293    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 03:19:18.478561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:18.478512    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:19:18.479087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:18.479070    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:19:18.479866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:18.479838    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:19:18.628285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:18.628235    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" podStartSLOduration=4.628220134 podStartE2EDuration="4.628220134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:16.213878037 +0000 UTC m=+706.228439758" watchObservedRunningTime="2026-10-31 03:19:18.628220134 +0000 UTC m=+708.642781841"
Oct 31 03:19:20.990751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.990708    5831 generic.go:356] "Generic (PLEG): container finished" podID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerID="8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf" exitCode=0
Oct 31 03:19:20.991202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.990774    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerDied","Data":"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"}
Oct 31 03:19:20.991202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.991138    5831 scope.go:122] "RemoveContainer" containerID="8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"
Oct 31 03:19:20.992470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.992443    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:20.992607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.992505    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:20.992651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:20.992613    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:22.015965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:22.015931    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerStarted","Data":"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"}
Oct 31 03:19:22.611391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:22.611353    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:19:22.839171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:22.839139    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:19:22.839376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:22.839269    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:19:22.839376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:22.839313    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 03:19:23.839420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:23.839371    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:19:23.840421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:23.839992    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 03:19:24.839597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:24.839553    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:19:24.840085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:24.839612    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 03:19:25.839347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:25.839312    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:19:25.839567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:25.839375    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 03:19:26.839288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:26.839254    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:19:26.839735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:26.839307    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 03:19:26.965240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:26.965208    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 03:19:26.965406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:26.965278    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 03:19:27.843889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:27.843853    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:19:27.844310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:27.844234    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:19:27.844882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:27.844864    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:19:32.018709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.018674    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:19:32.022832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.022808    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077571    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077607    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077729    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077796    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96kb\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077842    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.077991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.077939    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.078209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.078015    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.078209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.078044    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.119244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.119213    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:19:32.145669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.145596    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d39257-openshift-default-dockercfg-bm5z5\""
Oct 31 03:19:32.179168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179133    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179172    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179304    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179346    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179453    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179506    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179552    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179578    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179606    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179610    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96kb\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.179963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.179938    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.180131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.180108    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.180221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.180201    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.180288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.180207    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.182319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.182295    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.182929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.182911    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.232981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.232948    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.233202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.233182    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96kb\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.336788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.336701    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:32.830497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.830466    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:19:32.833946 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:19:32.833918    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b008e5f_6f87_4bba_80f1_a21818782fa9.slice/crio-8b667100bbb0437321fbbabd934b24b4a962ff036b9f96510c16b0e5c039935f WatchSource:0}: Error finding container 8b667100bbb0437321fbbabd934b24b4a962ff036b9f96510c16b0e5c039935f: Status 404 returned error can't find the container with id 8b667100bbb0437321fbbabd934b24b4a962ff036b9f96510c16b0e5c039935f
Oct 31 03:19:32.836221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.836188    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:32.836324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.836267    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:32.836324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:32.836316    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:19:33.102893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.102789    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" event={"ID":"5b008e5f-6f87-4bba-80f1-a21818782fa9","Type":"ContainerStarted","Data":"fd0b977931aebe8ba96ffbc7049600bc10e6f6dd0b2ceec4b516e93c52f6b299"}
Oct 31 03:19:33.102893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.102862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" event={"ID":"5b008e5f-6f87-4bba-80f1-a21818782fa9","Type":"ContainerStarted","Data":"8b667100bbb0437321fbbabd934b24b4a962ff036b9f96510c16b0e5c039935f"}
Oct 31 03:19:33.216605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.216524    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podStartSLOduration=2.216504145 podStartE2EDuration="2.216504145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:33.213268755 +0000 UTC m=+723.227830500" watchObservedRunningTime="2026-10-31 03:19:33.216504145 +0000 UTC m=+723.231065854"
Oct 31 03:19:33.337520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.337481    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:33.339050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.339023    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:33.339228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:33.339073    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:34.337106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:34.337061    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:34.337639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:34.337135    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:35.338050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:35.338016    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:35.338495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:35.338067    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:36.337239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:36.337205    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:36.337465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:36.337266    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:37.337910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:37.337873    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:37.338317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:37.337936    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:37.507186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:37.507151    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:19:38.337398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:38.337364    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:19:38.337593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:38.337420    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 03:19:39.341508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:39.341479    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:39.341915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:39.341831    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:39.342490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:39.342475    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:19:42.978155 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:19:42.978119    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:19:56.271501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:19:56.271464    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:20:12.991183 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:20:12.991141    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:20:14.689320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:20:14.689285    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:20:28.204747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:20:28.204711    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:20:42.002360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:20:42.002322    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:20:42.977322 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:20:42.977284    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:21:02.261231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:02.261140    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:21:06.252656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.252622    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 03:21:06.256693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.256674    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.353653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.353610    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"openshift-service-ca.crt\""
Oct 31 03:21:06.353653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.353633    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"kube-root-ca.crt\""
Oct 31 03:21:06.353897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.353875    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"default-dockercfg-w6g9t\""
Oct 31 03:21:06.353984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.353959    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"premium-w4-9360eb-kserve-self-signed-certs\""
Oct 31 03:21:06.356272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.356252    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 03:21:06.364351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364323    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.364461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364379    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.364560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364477    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.364560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364514    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.364690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364666    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.364745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.364698    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcl5r\" (UniqueName: \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465691    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465743    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465766    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465785    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465830    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.465927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.465849    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcl5r\" (UniqueName: \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.466195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.466113    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.466251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.466214    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.466297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.466266    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.468210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.468188    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.468444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.468427    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.519871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.519794    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcl5r\" (UniqueName: \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\") pod \"premium-w4-9360eb-kserve-5cd6f4bdc7-bs726\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") " pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.569168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.569139    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:06.970292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:06.970248    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 03:21:06.972962 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:21:06.972937    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4f15a43_2b9e_4b8e_88b2_33982e111aca.slice/crio-1f7fb5312e575066dee54d36a6b60553e64613bb698cadd3c3885ce5ad1759b7 WatchSource:0}: Error finding container 1f7fb5312e575066dee54d36a6b60553e64613bb698cadd3c3885ce5ad1759b7: Status 404 returned error can't find the container with id 1f7fb5312e575066dee54d36a6b60553e64613bb698cadd3c3885ce5ad1759b7
Oct 31 03:21:07.789876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:07.789840    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" event={"ID":"f4f15a43-2b9e-4b8e-88b2-33982e111aca","Type":"ContainerStarted","Data":"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"}
Oct 31 03:21:07.790381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:07.789890    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" event={"ID":"f4f15a43-2b9e-4b8e-88b2-33982e111aca","Type":"ContainerStarted","Data":"1f7fb5312e575066dee54d36a6b60553e64613bb698cadd3c3885ce5ad1759b7"}
Oct 31 03:21:07.900788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:07.900735    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podStartSLOduration=1.900721978 podStartE2EDuration="1.900721978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:07.898089728 +0000 UTC m=+817.912651433" watchObservedRunningTime="2026-10-31 03:21:07.900721978 +0000 UTC m=+817.915283685"
Oct 31 03:21:08.569975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:08.569939    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:12.993264 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:21:12.993190    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:21:16.569836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:16.569795    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:16.582835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:16.582806    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:16.862514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:16.862440    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:21:30.560561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:30.560512    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:21:42.979227 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:21:42.979186    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:21:51.808000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.807961    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"]
Oct 31 03:21:51.812137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.812116    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.864036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.864003    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:21:51.893655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.893620    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8kh4\" (UniqueName: \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.893836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.893801    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.893903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.893874    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.909677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.909653    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:21:51.909814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.909701    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:21:51.909814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.909766    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:21:51.912954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.912930    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"]
Oct 31 03:21:51.994606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.994568    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8kh4\" (UniqueName: \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.994818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.994656    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.994818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.994688    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.995311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.995290    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:51.997283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:51.997254    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:52.047572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.047510    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8kh4\" (UniqueName: \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\") pod \"maas-api-revoke-sub-0785a91e-tcg68\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:52.125269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.125196    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:52.239077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.239036    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 03:21:52.244297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.244168    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.293274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.293235    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"openshift-service-ca.crt\""
Oct 31 03:21:52.293597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.293266    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"kube-root-ca.crt\""
Oct 31 03:21:52.294899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.294274    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"default-dockercfg-7dmx6\""
Oct 31 03:21:52.295191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.295173    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"premium-w0-7778d0-kserve-self-signed-certs\""
Oct 31 03:21:52.298470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.298446    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 03:21:52.400117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400040    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7jd\" (UniqueName: \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.400267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400114    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.400267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400149    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.400345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400274    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.400345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400320    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.400419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.400402    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.499802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.499775    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"]
Oct 31 03:21:52.501083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501056    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501139    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501189    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501248    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501304    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7jd\" (UniqueName: \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501348    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501416    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501500    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.501987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.501964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.504234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.504212    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.504647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.504626    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.554808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.554779    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7jd\" (UniqueName: \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\") pod \"premium-w0-7778d0-kserve-6f78554b9c-zgb59\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") " pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.560948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.560926    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:52.617698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.615229    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:21:52.619697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:52.615595    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy" containerID="cri-o://26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37" gracePeriod=30
Oct 31 03:21:53.038586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:53.038553    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 03:21:53.052725 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:21:53.041787    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad31cd92_e0d4_4174_9f05_b2ab176b9036.slice/crio-8bef5cae5e3e80991f92de72af1c5dcd61dbda9ee73f70913c42edb99e261d77 WatchSource:0}: Error finding container 8bef5cae5e3e80991f92de72af1c5dcd61dbda9ee73f70913c42edb99e261d77: Status 404 returned error can't find the container with id 8bef5cae5e3e80991f92de72af1c5dcd61dbda9ee73f70913c42edb99e261d77
Oct 31 03:21:53.118427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:53.118396    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68" event={"ID":"4710e738-1fe7-43fc-977f-73920838acb9","Type":"ContainerStarted","Data":"73ba75af73a32c8295b6135fa93c825622de16acc3aef255b7152c5a98f55e20"}
Oct 31 03:21:53.118604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:53.118460    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68" event={"ID":"4710e738-1fe7-43fc-977f-73920838acb9","Type":"ContainerStarted","Data":"7c5da4966dd8e5db75e7377244e0dcc177e3a71277b9ebed41acdf4dcaf0fc0e"}
Oct 31 03:21:53.120601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:53.120572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" event={"ID":"ad31cd92-e0d4-4174-9f05-b2ab176b9036","Type":"ContainerStarted","Data":"8bef5cae5e3e80991f92de72af1c5dcd61dbda9ee73f70913c42edb99e261d77"}
Oct 31 03:21:53.223768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:53.223713    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68" podStartSLOduration=2.223698953 podStartE2EDuration="2.223698953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:53.222560465 +0000 UTC m=+863.237122170" watchObservedRunningTime="2026-10-31 03:21:53.223698953 +0000 UTC m=+863.238260661"
Oct 31 03:21:54.131646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:54.131615    5831 generic.go:356] "Generic (PLEG): container finished" podID="4710e738-1fe7-43fc-977f-73920838acb9" containerID="73ba75af73a32c8295b6135fa93c825622de16acc3aef255b7152c5a98f55e20" exitCode=0
Oct 31 03:21:54.136865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:54.136836    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68" event={"ID":"4710e738-1fe7-43fc-977f-73920838acb9","Type":"ContainerDied","Data":"73ba75af73a32c8295b6135fa93c825622de16acc3aef255b7152c5a98f55e20"}
Oct 31 03:21:54.137004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:54.136883    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" event={"ID":"ad31cd92-e0d4-4174-9f05-b2ab176b9036","Type":"ContainerStarted","Data":"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"}
Oct 31 03:21:54.347260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:54.347206    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podStartSLOduration=2.347192933 podStartE2EDuration="2.347192933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:54.346670309 +0000 UTC m=+864.361232017" watchObservedRunningTime="2026-10-31 03:21:54.347192933 +0000 UTC m=+864.361754640"
Oct 31 03:21:54.561215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:54.561179    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:21:55.292242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.291900    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:55.431889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.431861    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\") pod \"4710e738-1fe7-43fc-977f-73920838acb9\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") "
Oct 31 03:21:55.432098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.432050    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\" (UniqueName: \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\") pod \"4710e738-1fe7-43fc-977f-73920838acb9\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") "
Oct 31 03:21:55.432201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.432126    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\") pod \"4710e738-1fe7-43fc-977f-73920838acb9\" (UID: \"4710e738-1fe7-43fc-977f-73920838acb9\") "
Oct 31 03:21:55.432293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.432271    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca" pod "4710e738-1fe7-43fc-977f-73920838acb9" (UID: "4710e738-1fe7-43fc-977f-73920838acb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:55.432456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.432437    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:55.434454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.434425    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca" pod "4710e738-1fe7-43fc-977f-73920838acb9" (UID: "4710e738-1fe7-43fc-977f-73920838acb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:21:55.434625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.434515    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4" pod "4710e738-1fe7-43fc-977f-73920838acb9" (UID: "4710e738-1fe7-43fc-977f-73920838acb9"). InnerVolumeSpecName "kube-api-access-b8kh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:21:55.534064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.534028    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8kh4\" (UniqueName: \"kubernetes.io/projected/4710e738-1fe7-43fc-977f-73920838acb9-kube-api-access-b8kh4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:55.534064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:55.534058    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4710e738-1fe7-43fc-977f-73920838acb9-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:56.147427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:56.147399    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"
Oct 31 03:21:56.147584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:56.147435    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68" event={"ID":"4710e738-1fe7-43fc-977f-73920838acb9","Type":"ContainerDied","Data":"7c5da4966dd8e5db75e7377244e0dcc177e3a71277b9ebed41acdf4dcaf0fc0e"}
Oct 31 03:21:56.147584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:56.147487    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5da4966dd8e5db75e7377244e0dcc177e3a71277b9ebed41acdf4dcaf0fc0e"
Oct 31 03:21:57.931511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:57.931483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:21:58.061354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061326    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061380    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061492    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061510    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061565    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061605    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061655    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061688    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061768    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\") pod \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\" (UID: \"99e6e663-5b32-4d8c-a91a-fca269c69cc9\") "
Oct 31 03:21:58.061912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061846    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:58.062024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061912    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:58.062024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.061928    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:58.062326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062127    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:58.062326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062260    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.062326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062277    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.062767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062417    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:58.062868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062843    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.062950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.062880    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.063893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.063873    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "kube-api-access-mp2tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:21:58.064052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.064030    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:58.064362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.064337    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:21:58.064432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.064361    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token" pod "99e6e663-5b32-4d8c-a91a-fca269c69cc9" (UID: "99e6e663-5b32-4d8c-a91a-fca269c69cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:21:58.163762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.163688    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.163762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.163719    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.163762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.163732    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.163762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.163744    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6e663-5b32-4d8c-a91a-fca269c69cc9-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.163762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.163760    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2tv\" (UniqueName: \"kubernetes.io/projected/99e6e663-5b32-4d8c-a91a-fca269c69cc9-kube-api-access-mp2tv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:58.165731 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.165697    5831 generic.go:356] "Generic (PLEG): container finished" podID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerID="26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37" exitCode=0
Oct 31 03:21:58.165876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.165741    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerDied","Data":"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"}
Oct 31 03:21:58.165876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.165761    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"
Oct 31 03:21:58.165876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.165808    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr" event={"ID":"99e6e663-5b32-4d8c-a91a-fca269c69cc9","Type":"ContainerDied","Data":"970fa87d21f2efb8fd106e87a54243a0f3145829b20b17635b5ceab474aab8a4"}
Oct 31 03:21:58.165876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.165839    5831 scope.go:122] "RemoveContainer" containerID="26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"
Oct 31 03:21:58.176411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.176389    5831 scope.go:122] "RemoveContainer" containerID="3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"
Oct 31 03:21:58.186553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.186512    5831 scope.go:122] "RemoveContainer" containerID="26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"
Oct 31 03:21:58.186856 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:21:58.186829    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37\": container with ID starting with 26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37 not found: ID does not exist" containerID="26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"
Oct 31 03:21:58.186941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.186856    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37"} err="failed to get container status \"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37\": rpc error: code = NotFound desc = could not find container \"26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37\": container with ID starting with 26036a6889d34fbcd219c04de747c65f232915e499ed502825ae8b2f14f9ae37 not found: ID does not exist"
Oct 31 03:21:58.186941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.186880    5831 scope.go:122] "RemoveContainer" containerID="3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"
Oct 31 03:21:58.187182 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:21:58.187164    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c\": container with ID starting with 3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c not found: ID does not exist" containerID="3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"
Oct 31 03:21:58.187238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.187200    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c"} err="failed to get container status \"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c\": rpc error: code = NotFound desc = could not find container \"3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c\": container with ID starting with 3a6ad8c277c720ca7f99dc39ff750ee63a6dd7b9a515e9a849b9101a7895807c not found: ID does not exist"
Oct 31 03:21:58.330518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.330475    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:21:58.350155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:58.350126    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-22gdr"]
Oct 31 03:21:59.361145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:21:59.361103    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:22:00.173130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:00.173092    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" path="/var/lib/kubelet/pods/99e6e663-5b32-4d8c-a91a-fca269c69cc9/volumes"
Oct 31 03:22:02.561589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:02.561517    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:22:02.575708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:02.575676    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:22:03.238590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:03.238559    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:22:10.027479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.027440    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"]
Oct 31 03:22:10.028176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028153    5831 cpu_manager.go:414] "containerMap: removing container" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.028274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028178    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.028274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028212    5831 cpu_manager.go:414] "containerMap: removing container" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.028274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028220    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.028274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028242    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4710e738-1fe7-43fc-977f-73920838acb9" containerName="revoke-keys"
Oct 31 03:22:10.028274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028250    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4710e738-1fe7-43fc-977f-73920838acb9" containerName="revoke-keys"
Oct 31 03:22:10.028564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028386    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.028564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028401    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4710e738-1fe7-43fc-977f-73920838acb9" containerName="revoke-keys"
Oct 31 03:22:10.028564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.028418    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e6e663-5b32-4d8c-a91a-fca269c69cc9" containerName="istio-proxy"
Oct 31 03:22:10.032171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.032148    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.056837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.056813    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"]
Oct 31 03:22:10.084787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.084756    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:10.084942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.084802    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:10.084942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.084828    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:10.085086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.085064    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:10.119496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.119466    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clktc\" (UniqueName: \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.119677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.119558    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.119677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.119660    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.220901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.220866    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clktc\" (UniqueName: \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.221080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.221008    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.221080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.221073    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.221702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.221672    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.223597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.223575    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.273787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.273752    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clktc\" (UniqueName: \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\") pod \"maas-api-revoke-sub-feff57bd-4ngvz\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.345304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.345220    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:10.719576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:10.719409    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"]
Oct 31 03:22:10.722950 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:10.722922    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode100240e_a4fa_4a78_99eb_52e94e51c954.slice/crio-2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688 WatchSource:0}: Error finding container 2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688: Status 404 returned error can't find the container with id 2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688
Oct 31 03:22:11.091647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.091616    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"]
Oct 31 03:22:11.095891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.095869    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.131948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.131912    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.132104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.132078    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9gh\" (UniqueName: \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.132161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.132115    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.149865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.149836    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"]
Oct 31 03:22:11.233257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.233222    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9gh\" (UniqueName: \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.233451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.233270    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.233451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.233341    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.234232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.234196    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.236042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.236000    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.287066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.286987    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9gh\" (UniqueName: \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\") pod \"maas-api-revoke-sub-74021052-p474x\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.289668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.289629    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz" event={"ID":"e100240e-a4fa-4a78-99eb-52e94e51c954","Type":"ContainerStarted","Data":"4c6ea27b808dafceb284d9249aae9056c4a4a2729ed96403424d1086b711a38a"}
Oct 31 03:22:11.289797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.289700    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz" event={"ID":"e100240e-a4fa-4a78-99eb-52e94e51c954","Type":"ContainerStarted","Data":"2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688"}
Oct 31 03:22:11.401088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.401033    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz" podStartSLOduration=2.401018782 podStartE2EDuration="2.401018782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:11.398323693 +0000 UTC m=+881.412885401" watchObservedRunningTime="2026-10-31 03:22:11.401018782 +0000 UTC m=+881.415580490"
Oct 31 03:22:11.409616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.409582    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:11.782488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:11.782465    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"]
Oct 31 03:22:11.784647 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:11.784619    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77081a3b_8a48_49bf_b22d_985044fc3083.slice/crio-401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db WatchSource:0}: Error finding container 401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db: Status 404 returned error can't find the container with id 401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db
Oct 31 03:22:12.298752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:12.298721    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x" event={"ID":"77081a3b-8a48-49bf-b22d-985044fc3083","Type":"ContainerStarted","Data":"640135dc13f727fced192c0c88d6c95c059f44e6221daca5db77966b84854689"}
Oct 31 03:22:12.299227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:12.298773    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x" event={"ID":"77081a3b-8a48-49bf-b22d-985044fc3083","Type":"ContainerStarted","Data":"401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db"}
Oct 31 03:22:12.300676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:12.300642    5831 generic.go:356] "Generic (PLEG): container finished" podID="e100240e-a4fa-4a78-99eb-52e94e51c954" containerID="4c6ea27b808dafceb284d9249aae9056c4a4a2729ed96403424d1086b711a38a" exitCode=0
Oct 31 03:22:12.300818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:12.300690    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz" event={"ID":"e100240e-a4fa-4a78-99eb-52e94e51c954","Type":"ContainerDied","Data":"4c6ea27b808dafceb284d9249aae9056c4a4a2729ed96403424d1086b711a38a"}
Oct 31 03:22:12.406962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:12.406869    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x" podStartSLOduration=1.40685504 podStartE2EDuration="1.40685504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:12.405033946 +0000 UTC m=+882.419595654" watchObservedRunningTime="2026-10-31 03:22:12.40685504 +0000 UTC m=+882.421416745"
Oct 31 03:22:12.983215 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:22:12.983176    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:22:13.309644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.309510    5831 generic.go:356] "Generic (PLEG): container finished" podID="77081a3b-8a48-49bf-b22d-985044fc3083" containerID="640135dc13f727fced192c0c88d6c95c059f44e6221daca5db77966b84854689" exitCode=0
Oct 31 03:22:13.309644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.309598    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x" event={"ID":"77081a3b-8a48-49bf-b22d-985044fc3083","Type":"ContainerDied","Data":"640135dc13f727fced192c0c88d6c95c059f44e6221daca5db77966b84854689"}
Oct 31 03:22:13.456636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.456607    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:13.569020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.568930    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\" (UniqueName: \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\") pod \"e100240e-a4fa-4a78-99eb-52e94e51c954\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") "
Oct 31 03:22:13.569341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.569311    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\") pod \"e100240e-a4fa-4a78-99eb-52e94e51c954\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") "
Oct 31 03:22:13.569644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.569620    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\") pod \"e100240e-a4fa-4a78-99eb-52e94e51c954\" (UID: \"e100240e-a4fa-4a78-99eb-52e94e51c954\") "
Oct 31 03:22:13.571249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.571214    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca" pod "e100240e-a4fa-4a78-99eb-52e94e51c954" (UID: "e100240e-a4fa-4a78-99eb-52e94e51c954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:13.587908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.587359    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca" pod "e100240e-a4fa-4a78-99eb-52e94e51c954" (UID: "e100240e-a4fa-4a78-99eb-52e94e51c954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:13.588870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.588831    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc" pod "e100240e-a4fa-4a78-99eb-52e94e51c954" (UID: "e100240e-a4fa-4a78-99eb-52e94e51c954"). InnerVolumeSpecName "kube-api-access-clktc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:13.673569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.673520    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:13.673569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.673566    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clktc\" (UniqueName: \"kubernetes.io/projected/e100240e-a4fa-4a78-99eb-52e94e51c954-kube-api-access-clktc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:13.673569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:13.673577    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e100240e-a4fa-4a78-99eb-52e94e51c954-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:14.318932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.318896    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"
Oct 31 03:22:14.321026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.321000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz" event={"ID":"e100240e-a4fa-4a78-99eb-52e94e51c954","Type":"ContainerDied","Data":"2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688"}
Oct 31 03:22:14.321205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.321186    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2813dd9300f6b2b66b683bbdaa2612ee7ce58ca2e6c6727b331723fa127b1688"
Oct 31 03:22:14.488515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.488491    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:14.581845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.581814    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\") pod \"77081a3b-8a48-49bf-b22d-985044fc3083\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") "
Oct 31 03:22:14.582054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.581903    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\" (UniqueName: \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\") pod \"77081a3b-8a48-49bf-b22d-985044fc3083\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") "
Oct 31 03:22:14.582054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.581957    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\") pod \"77081a3b-8a48-49bf-b22d-985044fc3083\" (UID: \"77081a3b-8a48-49bf-b22d-985044fc3083\") "
Oct 31 03:22:14.582261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.582231    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca" pod "77081a3b-8a48-49bf-b22d-985044fc3083" (UID: "77081a3b-8a48-49bf-b22d-985044fc3083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:14.584215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.584179    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca" pod "77081a3b-8a48-49bf-b22d-985044fc3083" (UID: "77081a3b-8a48-49bf-b22d-985044fc3083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:14.584334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.584243    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh" pod "77081a3b-8a48-49bf-b22d-985044fc3083" (UID: "77081a3b-8a48-49bf-b22d-985044fc3083"). InnerVolumeSpecName "kube-api-access-mk9gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:14.683194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.683159    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:14.683194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.683191    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77081a3b-8a48-49bf-b22d-985044fc3083-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:14.683194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:14.683202    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9gh\" (UniqueName: \"kubernetes.io/projected/77081a3b-8a48-49bf-b22d-985044fc3083-kube-api-access-mk9gh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:15.328858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:15.328813    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x" event={"ID":"77081a3b-8a48-49bf-b22d-985044fc3083","Type":"ContainerDied","Data":"401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db"}
Oct 31 03:22:15.328858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:15.328835    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"
Oct 31 03:22:15.328858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:15.328865    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401058c06fd0ff65399e84f57ebe7e5f7f67cc27817cf05b7d6950f8762e47db"
Oct 31 03:22:20.375593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.375559    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"]
Oct 31 03:22:20.376260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376239    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e100240e-a4fa-4a78-99eb-52e94e51c954" containerName="revoke-keys"
Oct 31 03:22:20.376327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376264    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e100240e-a4fa-4a78-99eb-52e94e51c954" containerName="revoke-keys"
Oct 31 03:22:20.376327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376315    5831 cpu_manager.go:414] "containerMap: removing container" podUID="77081a3b-8a48-49bf-b22d-985044fc3083" containerName="revoke-keys"
Oct 31 03:22:20.376327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376323    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77081a3b-8a48-49bf-b22d-985044fc3083" containerName="revoke-keys"
Oct 31 03:22:20.376497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376476    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e100240e-a4fa-4a78-99eb-52e94e51c954" containerName="revoke-keys"
Oct 31 03:22:20.376497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.376497    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="77081a3b-8a48-49bf-b22d-985044fc3083" containerName="revoke-keys"
Oct 31 03:22:20.380089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.380071    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.430500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.430471    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:20.441175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.441139    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblph\" (UniqueName: \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.441350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.441196    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.441350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.441321    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.476358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.476324    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:20.476358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.476352    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:20.476569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.476373    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:20.479435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.479415    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"]
Oct 31 03:22:20.542752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.542720    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.542927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.542777    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.542927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.542856    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblph\" (UniqueName: \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.543565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.543518    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.545291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.545274    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.604017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.603986    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblph\" (UniqueName: \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\") pod \"maas-api-revoke-sub-16c1656d-vgcxp\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:20.694157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:20.694124    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:21.066020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:21.065995    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"]
Oct 31 03:22:21.067786 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:21.067747    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d077e2_f93c_4ec6_9aa8_3340c08c804b.slice/crio-d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f WatchSource:0}: Error finding container d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f: Status 404 returned error can't find the container with id d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f
Oct 31 03:22:21.369928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:21.369837    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp" event={"ID":"52d077e2-f93c-4ec6-9aa8-3340c08c804b","Type":"ContainerStarted","Data":"d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f"}
Oct 31 03:22:22.385606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:22.385191    5831 generic.go:356] "Generic (PLEG): container finished" podID="52d077e2-f93c-4ec6-9aa8-3340c08c804b" containerID="8e0a30bc75c4de2463277b16d8a65e4ac58dcd6abf4c5e3aa82dfea89fe7a0b7" exitCode=0
Oct 31 03:22:22.385606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:22.385324    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp" event={"ID":"52d077e2-f93c-4ec6-9aa8-3340c08c804b","Type":"ContainerDied","Data":"8e0a30bc75c4de2463277b16d8a65e4ac58dcd6abf4c5e3aa82dfea89fe7a0b7"}
Oct 31 03:22:23.548148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.548121    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:23.683740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.683709    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\") pod \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") "
Oct 31 03:22:23.683940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.683851    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\" (UniqueName: \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\") pod \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") "
Oct 31 03:22:23.683940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.683896    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\") pod \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\" (UID: \"52d077e2-f93c-4ec6-9aa8-3340c08c804b\") "
Oct 31 03:22:23.684183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.684154    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca" pod "52d077e2-f93c-4ec6-9aa8-3340c08c804b" (UID: "52d077e2-f93c-4ec6-9aa8-3340c08c804b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:23.684322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.684304    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:23.686962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.686929    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph" pod "52d077e2-f93c-4ec6-9aa8-3340c08c804b" (UID: "52d077e2-f93c-4ec6-9aa8-3340c08c804b"). InnerVolumeSpecName "kube-api-access-xblph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:23.687108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.686985    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca" pod "52d077e2-f93c-4ec6-9aa8-3340c08c804b" (UID: "52d077e2-f93c-4ec6-9aa8-3340c08c804b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:23.785224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.785192    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblph\" (UniqueName: \"kubernetes.io/projected/52d077e2-f93c-4ec6-9aa8-3340c08c804b-kube-api-access-xblph\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:23.785224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:23.785222    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d077e2-f93c-4ec6-9aa8-3340c08c804b-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:24.401140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:24.401069    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"
Oct 31 03:22:24.401276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:24.401062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp" event={"ID":"52d077e2-f93c-4ec6-9aa8-3340c08c804b","Type":"ContainerDied","Data":"d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f"}
Oct 31 03:22:24.401276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:24.401249    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29b0f853f753c6f5ecffe8bc854c02e19b90768b4df4735f0baf078370d422f"
Oct 31 03:22:26.440353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.440318    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"]
Oct 31 03:22:26.440882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.440868    5831 cpu_manager.go:414] "containerMap: removing container" podUID="52d077e2-f93c-4ec6-9aa8-3340c08c804b" containerName="revoke-keys"
Oct 31 03:22:26.440936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.440883    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d077e2-f93c-4ec6-9aa8-3340c08c804b" containerName="revoke-keys"
Oct 31 03:22:26.440986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.440975    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d077e2-f93c-4ec6-9aa8-3340c08c804b" containerName="revoke-keys"
Oct 31 03:22:26.444798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.444778    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.498258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.498226    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"]
Oct 31 03:22:26.516482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.516443    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntxc\" (UniqueName: \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.516698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.516509    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.516698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.516659    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.539967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.539926    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:26.539967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.539931    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:26.539967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.539939    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:26.540192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.539961    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:26.617722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.617683    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.617897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.617784    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntxc\" (UniqueName: \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.617897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.617820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.618483 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.618457    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.620191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.620167    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.670963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.670922    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntxc\" (UniqueName: \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\") pod \"maas-api-revoke-sub-b8013e78-96xsf\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:26.761777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:26.761698    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:27.139805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:27.139782    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"]
Oct 31 03:22:27.142444 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:27.142409    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5796bd11_b982_4c62_a657_a9b7fab3d829.slice/crio-9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a WatchSource:0}: Error finding container 9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a: Status 404 returned error can't find the container with id 9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a
Oct 31 03:22:27.434494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:27.434461    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf" event={"ID":"5796bd11-b982-4c62-a657-a9b7fab3d829","Type":"ContainerStarted","Data":"467d2e7df09e05935056f3f3735b0147b93074a6998e17494d1052aef474bc50"}
Oct 31 03:22:27.434696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:27.434512    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf" event={"ID":"5796bd11-b982-4c62-a657-a9b7fab3d829","Type":"ContainerStarted","Data":"9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a"}
Oct 31 03:22:27.543506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:27.543454    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf" podStartSLOduration=1.5434388220000002 podStartE2EDuration="1.543438822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:27.54018133 +0000 UTC m=+897.554743037" watchObservedRunningTime="2026-10-31 03:22:27.543438822 +0000 UTC m=+897.558000530"
Oct 31 03:22:28.443078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:28.443050    5831 generic.go:356] "Generic (PLEG): container finished" podID="5796bd11-b982-4c62-a657-a9b7fab3d829" containerID="467d2e7df09e05935056f3f3735b0147b93074a6998e17494d1052aef474bc50" exitCode=0
Oct 31 03:22:28.443248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:28.443135    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf" event={"ID":"5796bd11-b982-4c62-a657-a9b7fab3d829","Type":"ContainerDied","Data":"467d2e7df09e05935056f3f3735b0147b93074a6998e17494d1052aef474bc50"}
Oct 31 03:22:28.955283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:28.955250    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:22:29.593434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.593405    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:29.755313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.755231    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\" (UniqueName: \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\") pod \"5796bd11-b982-4c62-a657-a9b7fab3d829\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") "
Oct 31 03:22:29.755313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.755294    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\") pod \"5796bd11-b982-4c62-a657-a9b7fab3d829\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") "
Oct 31 03:22:29.755579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.755379    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\") pod \"5796bd11-b982-4c62-a657-a9b7fab3d829\" (UID: \"5796bd11-b982-4c62-a657-a9b7fab3d829\") "
Oct 31 03:22:29.755753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.755729    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca" pod "5796bd11-b982-4c62-a657-a9b7fab3d829" (UID: "5796bd11-b982-4c62-a657-a9b7fab3d829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:29.755854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.755835    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:29.757685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.757655    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca" pod "5796bd11-b982-4c62-a657-a9b7fab3d829" (UID: "5796bd11-b982-4c62-a657-a9b7fab3d829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:29.757787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.757758    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc" pod "5796bd11-b982-4c62-a657-a9b7fab3d829" (UID: "5796bd11-b982-4c62-a657-a9b7fab3d829"). InnerVolumeSpecName "kube-api-access-dntxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:29.857168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.857119    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dntxc\" (UniqueName: \"kubernetes.io/projected/5796bd11-b982-4c62-a657-a9b7fab3d829-kube-api-access-dntxc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:29.857168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:29.857156    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796bd11-b982-4c62-a657-a9b7fab3d829-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:30.260976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.260947    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:22:30.263386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.263365    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:22:30.272221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.272196    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:22:30.273999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.273968    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:22:30.276284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.276267    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:22:30.277971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.277950    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:22:30.457995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.457961    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"
Oct 31 03:22:30.458249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.458008    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf" event={"ID":"5796bd11-b982-4c62-a657-a9b7fab3d829","Type":"ContainerDied","Data":"9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a"}
Oct 31 03:22:30.458249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:30.458061    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9adbe0ebbbe968052db2244ce7e72cd84adc1b6ff17b182784aaed01c20ab71a"
Oct 31 03:22:36.273262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.273221    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"]
Oct 31 03:22:36.274008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.273987    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5796bd11-b982-4c62-a657-a9b7fab3d829" containerName="revoke-keys"
Oct 31 03:22:36.274099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.274010    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5796bd11-b982-4c62-a657-a9b7fab3d829" containerName="revoke-keys"
Oct 31 03:22:36.274157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.274141    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5796bd11-b982-4c62-a657-a9b7fab3d829" containerName="revoke-keys"
Oct 31 03:22:36.277838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.277815    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.329695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.329666    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"]
Oct 31 03:22:36.372132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.372102    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:36.372297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.372152    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:36.372297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.372229    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:36.372297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.372244    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:36.431619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.431585    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.431796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.431725    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.431796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.431757    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqr2\" (UniqueName: \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.533475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.533385    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.533475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.533421    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqr2\" (UniqueName: \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.533713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.533504    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.534201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.534177    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.536062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.536034    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.588878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.588848    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqr2\" (UniqueName: \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\") pod \"maas-api-revoke-sub-daa76307-h5k6z\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.592570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.592519    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:36.604874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.604850    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 03:22:36.608944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.608927    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.660801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.660748    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 03:22:36.665799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.665768    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 03:22:36.735439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735409    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735461    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735485    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735637    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735661    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735680    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735795    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735828    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.735945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.735865    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837103    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837183    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837243    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837303    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837367    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837398    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837436    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837554    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837599    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.837862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837702    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.838291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837938    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.838291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.837996    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.838291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.838189    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.838291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.838238    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.840233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.840208    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.840347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.840316    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.891151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.891121    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.891767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.891742    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.935509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.935485    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:36.965823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:36.965791    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"]
Oct 31 03:22:36.969585 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:36.969554    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436ec380_7dcb_43d5_a75b_70b7108041bc.slice/crio-da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb WatchSource:0}: Error finding container da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb: Status 404 returned error can't find the container with id da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb
Oct 31 03:22:37.313256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.313233    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 03:22:37.315148 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:37.315120    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb143617b_97cc_4b31_be9e_725c3efb4651.slice/crio-bc22dee7de7fd3cfd3588e52c3b3820f55bb00b4ba5cc4fe813d0b9a1d0067ba WatchSource:0}: Error finding container bc22dee7de7fd3cfd3588e52c3b3820f55bb00b4ba5cc4fe813d0b9a1d0067ba: Status 404 returned error can't find the container with id bc22dee7de7fd3cfd3588e52c3b3820f55bb00b4ba5cc4fe813d0b9a1d0067ba
Oct 31 03:22:37.317639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.317610    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:22:37.317725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.317683    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:22:37.317725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.317714    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:22:37.516070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.515961    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" event={"ID":"b143617b-97cc-4b31-be9e-725c3efb4651","Type":"ContainerStarted","Data":"f633b6610175a025e658abf5366142b384263ead006b8947095f84721ddeb2c5"}
Oct 31 03:22:37.516250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.516072    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" event={"ID":"b143617b-97cc-4b31-be9e-725c3efb4651","Type":"ContainerStarted","Data":"bc22dee7de7fd3cfd3588e52c3b3820f55bb00b4ba5cc4fe813d0b9a1d0067ba"}
Oct 31 03:22:37.519463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.519428    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z" event={"ID":"436ec380-7dcb-43d5-a75b-70b7108041bc","Type":"ContainerStarted","Data":"8d8e8af8da2f1ce206e3b37f4f8fa76a78257e695eed1a164b0f3b54c59b710e"}
Oct 31 03:22:37.519613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.519487    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z" event={"ID":"436ec380-7dcb-43d5-a75b-70b7108041bc","Type":"ContainerStarted","Data":"da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb"}
Oct 31 03:22:37.631923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.631855    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podStartSLOduration=1.631836919 podStartE2EDuration="1.631836919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:37.628576381 +0000 UTC m=+907.643138084" watchObservedRunningTime="2026-10-31 03:22:37.631836919 +0000 UTC m=+907.646398628"
Oct 31 03:22:37.738540 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.738474    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z" podStartSLOduration=1.7384605899999999 podStartE2EDuration="1.73846059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:37.735936165 +0000 UTC m=+907.750497875" watchObservedRunningTime="2026-10-31 03:22:37.73846059 +0000 UTC m=+907.753022297"
Oct 31 03:22:37.936133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:37.936101    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:38.528945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:38.528912    5831 generic.go:356] "Generic (PLEG): container finished" podID="436ec380-7dcb-43d5-a75b-70b7108041bc" containerID="8d8e8af8da2f1ce206e3b37f4f8fa76a78257e695eed1a164b0f3b54c59b710e" exitCode=0
Oct 31 03:22:38.529404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:38.528995    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z" event={"ID":"436ec380-7dcb-43d5-a75b-70b7108041bc","Type":"ContainerDied","Data":"8d8e8af8da2f1ce206e3b37f4f8fa76a78257e695eed1a164b0f3b54c59b710e"}
Oct 31 03:22:38.938342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:38.938301    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:22:38.938519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:38.938364    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:22:39.745365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.745339    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:39.871205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.871126    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\") pod \"436ec380-7dcb-43d5-a75b-70b7108041bc\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") "
Oct 31 03:22:39.871348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.871230    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\") pod \"436ec380-7dcb-43d5-a75b-70b7108041bc\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") "
Oct 31 03:22:39.871348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.871257    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\" (UniqueName: \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\") pod \"436ec380-7dcb-43d5-a75b-70b7108041bc\" (UID: \"436ec380-7dcb-43d5-a75b-70b7108041bc\") "
Oct 31 03:22:39.871546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.871506    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca" pod "436ec380-7dcb-43d5-a75b-70b7108041bc" (UID: "436ec380-7dcb-43d5-a75b-70b7108041bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:39.871718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.871694    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:39.873448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.873420    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca" pod "436ec380-7dcb-43d5-a75b-70b7108041bc" (UID: "436ec380-7dcb-43d5-a75b-70b7108041bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:39.873550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.873484    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2" pod "436ec380-7dcb-43d5-a75b-70b7108041bc" (UID: "436ec380-7dcb-43d5-a75b-70b7108041bc"). InnerVolumeSpecName "kube-api-access-wkqr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:39.936319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.936284    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:22:39.936480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.936339    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:22:39.972668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.972642    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436ec380-7dcb-43d5-a75b-70b7108041bc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:39.972668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:39.972668    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqr2\" (UniqueName: \"kubernetes.io/projected/436ec380-7dcb-43d5-a75b-70b7108041bc-kube-api-access-wkqr2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:40.543849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.543816    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z" event={"ID":"436ec380-7dcb-43d5-a75b-70b7108041bc","Type":"ContainerDied","Data":"da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb"}
Oct 31 03:22:40.543849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.543846    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"
Oct 31 03:22:40.544063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.543866    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da5d3d8d552b8039c2e1e64b396da978d4dde4c9c429e6800c28ca7e699230cb"
Oct 31 03:22:40.705803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.705774    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"]
Oct 31 03:22:40.706284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.706268    5831 cpu_manager.go:414] "containerMap: removing container" podUID="436ec380-7dcb-43d5-a75b-70b7108041bc" containerName="revoke-keys"
Oct 31 03:22:40.706284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.706282    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436ec380-7dcb-43d5-a75b-70b7108041bc" containerName="revoke-keys"
Oct 31 03:22:40.706473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.706381    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="436ec380-7dcb-43d5-a75b-70b7108041bc" containerName="revoke-keys"
Oct 31 03:22:40.710126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.710101    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.755105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.755078    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"]
Oct 31 03:22:40.759919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.759897    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:40.809889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.809790    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:40.810076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.809896    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:40.810076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.809964    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:40.810346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.810325    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbjz\" (UniqueName: \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.810492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.810474    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.810606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.810512    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.911125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.911090    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.911326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.911136    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.911326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.911199    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbjz\" (UniqueName: \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.911841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.911814    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.913736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.913713    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:40.935835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.935809    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:22:40.935944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.935855    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:22:40.964628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:40.964606    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbjz\" (UniqueName: \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\") pod \"maas-api-revoke-sub-e16d4902-22jfb\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:41.024841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:41.024804    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:41.436939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:41.436913    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"]
Oct 31 03:22:41.437716 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:41.437679    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95792043_3c91_452d_baab_438f2de0fedf.slice/crio-54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d WatchSource:0}: Error finding container 54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d: Status 404 returned error can't find the container with id 54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d
Oct 31 03:22:41.551442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:41.551407    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb" event={"ID":"95792043-3c91-452d-baab-438f2de0fedf","Type":"ContainerStarted","Data":"54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d"}
Oct 31 03:22:41.936714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:41.936682    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:22:41.937042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:41.936728    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:22:42.560581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:42.560516    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb" event={"ID":"95792043-3c91-452d-baab-438f2de0fedf","Type":"ContainerStarted","Data":"ced7cfd9f72e44c6a4e8ee64fe877dbdc5584d33f84181cf457e07b5cd0b395f"}
Oct 31 03:22:42.667335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:42.667282    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb" podStartSLOduration=2.667266972 podStartE2EDuration="2.667266972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:42.666019294 +0000 UTC m=+912.680581004" watchObservedRunningTime="2026-10-31 03:22:42.667266972 +0000 UTC m=+912.681828680"
Oct 31 03:22:42.936090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:42.936060    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:22:42.936272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:42.936109    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:22:42.987934 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:22:42.987900    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:22:43.568815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:43.568779    5831 generic.go:356] "Generic (PLEG): container finished" podID="95792043-3c91-452d-baab-438f2de0fedf" containerID="ced7cfd9f72e44c6a4e8ee64fe877dbdc5584d33f84181cf457e07b5cd0b395f" exitCode=0
Oct 31 03:22:43.569024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:43.568876    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb" event={"ID":"95792043-3c91-452d-baab-438f2de0fedf","Type":"ContainerDied","Data":"ced7cfd9f72e44c6a4e8ee64fe877dbdc5584d33f84181cf457e07b5cd0b395f"}
Oct 31 03:22:44.123467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.122241    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:44.124000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.123661    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:44.124628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.124608    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 03:22:44.715649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.715623    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:44.763149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.763117    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\") pod \"95792043-3c91-452d-baab-438f2de0fedf\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") "
Oct 31 03:22:44.763303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.763253    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\") pod \"95792043-3c91-452d-baab-438f2de0fedf\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") "
Oct 31 03:22:44.763341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.763301    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\" (UniqueName: \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\") pod \"95792043-3c91-452d-baab-438f2de0fedf\" (UID: \"95792043-3c91-452d-baab-438f2de0fedf\") "
Oct 31 03:22:44.763608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.763581    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca" pod "95792043-3c91-452d-baab-438f2de0fedf" (UID: "95792043-3c91-452d-baab-438f2de0fedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:44.763760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.763682    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95792043-3c91-452d-baab-438f2de0fedf-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:44.765441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.765414    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca" pod "95792043-3c91-452d-baab-438f2de0fedf" (UID: "95792043-3c91-452d-baab-438f2de0fedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:44.765701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.765680    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz" pod "95792043-3c91-452d-baab-438f2de0fedf" (UID: "95792043-3c91-452d-baab-438f2de0fedf"). InnerVolumeSpecName "kube-api-access-hgbjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:44.864443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.864365    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbjz\" (UniqueName: \"kubernetes.io/projected/95792043-3c91-452d-baab-438f2de0fedf-kube-api-access-hgbjz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:44.864443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:44.864391    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95792043-3c91-452d-baab-438f2de0fedf-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:45.584259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:45.584222    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"
Oct 31 03:22:45.584799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:45.584250    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb" event={"ID":"95792043-3c91-452d-baab-438f2de0fedf","Type":"ContainerDied","Data":"54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d"}
Oct 31 03:22:45.584799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:45.584309    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54cea0e3a21669a4e4269cb683e01db2b4cbc5c1b091773f3caedb5b3d914b2d"
Oct 31 03:22:47.192381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.192318    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:22:47.193255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.193233    5831 cpu_manager.go:414] "containerMap: removing container" podUID="95792043-3c91-452d-baab-438f2de0fedf" containerName="revoke-keys"
Oct 31 03:22:47.193354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.193257    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95792043-3c91-452d-baab-438f2de0fedf" containerName="revoke-keys"
Oct 31 03:22:47.193466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.193451    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="95792043-3c91-452d-baab-438f2de0fedf" containerName="revoke-keys"
Oct 31 03:22:47.198292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.198267    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.266370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.258282    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:22:47.288347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.288315    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.288347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.288353    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fxg\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.288661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.288407    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.288661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.288444    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.294745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.294712    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xzl94\""
Oct 31 03:22:47.294893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.294805    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a4124b3dfa38\""
Oct 31 03:22:47.294893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.294843    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-280afa1d2683\""
Oct 31 03:22:47.295543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.295512    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 03:22:47.295687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.295673    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 03:22:47.295730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.295718    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 03:22:47.295824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.295811    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 03:22:47.389193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.389142    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.389376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.389204    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fxg\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.389376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.389311    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.389522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.389377    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.390121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.390096    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.390308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.390281    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.392298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.392270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.442738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.442666    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fxg\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\") pod \"payload-processing-external-model-5b7c6c55f6-7m6tv\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") " pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.518887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.518844    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:47.904456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:47.904423    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:22:47.906117 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:22:47.906074    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b235195_3c14_4aea_af46_cd14114ac5e5.slice/crio-dd55f20563111969f733e5d1bf48c28031e5f375fb86641a8c44df6f1c85345a WatchSource:0}: Error finding container dd55f20563111969f733e5d1bf48c28031e5f375fb86641a8c44df6f1c85345a: Status 404 returned error can't find the container with id dd55f20563111969f733e5d1bf48c28031e5f375fb86641a8c44df6f1c85345a
Oct 31 03:22:48.644567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:48.633652    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" event={"ID":"8b235195-3c14-4aea-af46-cd14114ac5e5","Type":"ContainerStarted","Data":"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"}
Oct 31 03:22:48.645290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:48.645210    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" event={"ID":"8b235195-3c14-4aea-af46-cd14114ac5e5","Type":"ContainerStarted","Data":"dd55f20563111969f733e5d1bf48c28031e5f375fb86641a8c44df6f1c85345a"}
Oct 31 03:22:48.646442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:48.646387    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:22:48.776233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:48.776172    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" podStartSLOduration=1.776151128 podStartE2EDuration="1.776151128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:48.77399341 +0000 UTC m=+918.788555147" watchObservedRunningTime="2026-10-31 03:22:48.776151128 +0000 UTC m=+918.790712837"
Oct 31 03:22:51.463887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.463852    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"]
Oct 31 03:22:51.468099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.468078    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.539573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.530905    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"]
Oct 31 03:22:51.545751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.545712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.545875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.545765    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhcq\" (UniqueName: \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.545952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.545934    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.564547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.564496    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:22:51.564694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.564559    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:22:51.564694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.564576    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:22:51.564694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.564617    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:22:51.647211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.647166    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.647389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.647255    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.647389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.647292    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhcq\" (UniqueName: \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.647974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.647949    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.649760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.649731    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.703181 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.703146    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhcq\" (UniqueName: \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\") pod \"maas-api-revoke-sub-e5cedb47-mgxft\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:51.830376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:51.829735    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:52.277069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:52.273433    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"]
Oct 31 03:22:52.665980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:52.665939    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft" event={"ID":"4239c725-9830-4494-b4b2-2f6a6b50af58","Type":"ContainerStarted","Data":"1e61df17cb62e52c6a5f9e284750fe5433f79c246624dcaf5c4eabab2b9a4155"}
Oct 31 03:22:52.666371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:52.665992    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft" event={"ID":"4239c725-9830-4494-b4b2-2f6a6b50af58","Type":"ContainerStarted","Data":"8a830e078f3c713e20b9b1b078f5e5d9d05e9d50d8fbd29469f878e148609355"}
Oct 31 03:22:52.793170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:52.793117    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft" podStartSLOduration=1.793102107 podStartE2EDuration="1.793102107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:52.789084736 +0000 UTC m=+922.803646444" watchObservedRunningTime="2026-10-31 03:22:52.793102107 +0000 UTC m=+922.807663815"
Oct 31 03:22:53.675057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:53.674961    5831 generic.go:356] "Generic (PLEG): container finished" podID="4239c725-9830-4494-b4b2-2f6a6b50af58" containerID="1e61df17cb62e52c6a5f9e284750fe5433f79c246624dcaf5c4eabab2b9a4155" exitCode=0
Oct 31 03:22:53.675057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:53.675029    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft" event={"ID":"4239c725-9830-4494-b4b2-2f6a6b50af58","Type":"ContainerDied","Data":"1e61df17cb62e52c6a5f9e284750fe5433f79c246624dcaf5c4eabab2b9a4155"}
Oct 31 03:22:54.839445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.839421    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:54.894295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.894268    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\" (UniqueName: \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\") pod \"4239c725-9830-4494-b4b2-2f6a6b50af58\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") "
Oct 31 03:22:54.894458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.894329    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\") pod \"4239c725-9830-4494-b4b2-2f6a6b50af58\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") "
Oct 31 03:22:54.894458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.894401    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\") pod \"4239c725-9830-4494-b4b2-2f6a6b50af58\" (UID: \"4239c725-9830-4494-b4b2-2f6a6b50af58\") "
Oct 31 03:22:54.894702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.894681    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca" pod "4239c725-9830-4494-b4b2-2f6a6b50af58" (UID: "4239c725-9830-4494-b4b2-2f6a6b50af58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:22:54.894789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.894773    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:54.896592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.896571    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca" pod "4239c725-9830-4494-b4b2-2f6a6b50af58" (UID: "4239c725-9830-4494-b4b2-2f6a6b50af58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:22:54.896668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.896653    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq" pod "4239c725-9830-4494-b4b2-2f6a6b50af58" (UID: "4239c725-9830-4494-b4b2-2f6a6b50af58"). InnerVolumeSpecName "kube-api-access-bbhcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:22:54.996037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.995958    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhcq\" (UniqueName: \"kubernetes.io/projected/4239c725-9830-4494-b4b2-2f6a6b50af58-kube-api-access-bbhcq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:54.996037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:54.995985    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4239c725-9830-4494-b4b2-2f6a6b50af58-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:22:55.697781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:55.697750    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"
Oct 31 03:22:55.697985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:55.697742    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft" event={"ID":"4239c725-9830-4494-b4b2-2f6a6b50af58","Type":"ContainerDied","Data":"8a830e078f3c713e20b9b1b078f5e5d9d05e9d50d8fbd29469f878e148609355"}
Oct 31 03:22:55.697985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:22:55.697906    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a830e078f3c713e20b9b1b078f5e5d9d05e9d50d8fbd29469f878e148609355"
Oct 31 03:23:00.653092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:00.653066    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:23:02.662992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:02.662956    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:23:08.186214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.186180    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:23:08.186949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.186927    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4239c725-9830-4494-b4b2-2f6a6b50af58" containerName="revoke-keys"
Oct 31 03:23:08.187052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.186951    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4239c725-9830-4494-b4b2-2f6a6b50af58" containerName="revoke-keys"
Oct 31 03:23:08.187118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.187105    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4239c725-9830-4494-b4b2-2f6a6b50af58" containerName="revoke-keys"
Oct 31 03:23:08.191133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.191109    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.228925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.228890    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:23:08.233685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.233663    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.269299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269255    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269449    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x6r\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269494    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269592    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269668    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269702    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269851    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.269936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.269901    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.270076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.270051    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.277814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.277790    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:23:08.354813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.354772    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373127    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373194    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373230    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373257    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373286    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373321    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373351    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373376    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373411    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373444    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv44h\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373499    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373553    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373628    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373667    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373691    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373718    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x6r\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.373821    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.374649    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.384633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.377450    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.384633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.380664    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.384633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.380944    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.385328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.385300    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.414908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.414866    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.422655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.422625    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.438451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.438378    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.438451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.438381    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x6r\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\") pod \"maas-default-gateway-openshift-default-6886b96f9-h2zq7\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.475361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475323    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475370    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475387    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475403    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475491    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475518    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475558    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475573    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475605    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv44h\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.475902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.475877    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.476184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.476163    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.476303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.476231    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.476488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.476461    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.476659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.476639    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.479021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.478992    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.479230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.479209    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.514442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.514417    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:08.531151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.531132    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.532468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.532446    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv44h\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\") pod \"maas-default-gateway-openshift-default-6886b96f9-zp2v6\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.548406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.548378    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:08.940499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.940470    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:23:08.942162 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:23:08.942135    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37340dc1_dbe9_4b25_892b_1d44638c5b3f.slice/crio-57ddae0dd3b38c35c074262b5022ba3a54175112fcf8c6ef95479200fa91b4a6 WatchSource:0}: Error finding container 57ddae0dd3b38c35c074262b5022ba3a54175112fcf8c6ef95479200fa91b4a6: Status 404 returned error can't find the container with id 57ddae0dd3b38c35c074262b5022ba3a54175112fcf8c6ef95479200fa91b4a6
Oct 31 03:23:08.944201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.944179    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:23:08.944914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.944887    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:08.944989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.944950    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:08.944989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.944979    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:08.976896 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:23:08.976873    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74d7f77_bbea_4ca9_b6e4_c60078c63e7d.slice/crio-2fd8bed6f1ab9e61293221cbc016d3da6a748fe844e4a2e6df8cc15edb5c2051 WatchSource:0}: Error finding container 2fd8bed6f1ab9e61293221cbc016d3da6a748fe844e4a2e6df8cc15edb5c2051: Status 404 returned error can't find the container with id 2fd8bed6f1ab9e61293221cbc016d3da6a748fe844e4a2e6df8cc15edb5c2051
Oct 31 03:23:08.979805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.979772    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:08.979913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.979838    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:08.979913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:08.979870    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:09.413424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.413345    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:09.418408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.418382    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.470367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.470335    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:09.486987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.486950    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.487160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.487039    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.487213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.487167    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv2m\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.487249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.487209    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.589506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.589446    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.589912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.589882    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.590091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.590072    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv2m\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.590242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.590225    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.591393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.591372    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.624370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.624332    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.643279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.641503    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.722555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.722239    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv2m\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\") pod \"payload-processing-external-model-54b7cc8547-64sbm\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") " pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.739958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.739917    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:09.835515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.835467    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" event={"ID":"37340dc1-dbe9-4b25-892b-1d44638c5b3f","Type":"ContainerStarted","Data":"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"}
Oct 31 03:23:09.835876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.835852    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" event={"ID":"37340dc1-dbe9-4b25-892b-1d44638c5b3f","Type":"ContainerStarted","Data":"57ddae0dd3b38c35c074262b5022ba3a54175112fcf8c6ef95479200fa91b4a6"}
Oct 31 03:23:09.839732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.839693    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" event={"ID":"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d","Type":"ContainerStarted","Data":"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"}
Oct 31 03:23:09.839855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:09.839751    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" event={"ID":"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d","Type":"ContainerStarted","Data":"2fd8bed6f1ab9e61293221cbc016d3da6a748fe844e4a2e6df8cc15edb5c2051"}
Oct 31 03:23:10.126550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.126404    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podStartSLOduration=2.126386122 podStartE2EDuration="2.126386122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:10.010883196 +0000 UTC m=+940.025444942" watchObservedRunningTime="2026-10-31 03:23:10.126386122 +0000 UTC m=+940.140947832"
Oct 31 03:23:10.131414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.131354    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podStartSLOduration=2.131334308 podStartE2EDuration="2.131334308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:10.124416278 +0000 UTC m=+940.138977990" watchObservedRunningTime="2026-10-31 03:23:10.131334308 +0000 UTC m=+940.145896018"
Oct 31 03:23:10.163782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.163758    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:10.517983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.517901    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:10.528010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.527660    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:10.528010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.527723    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:10.548873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.548808    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:10.550750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.550662    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:10.550750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.550706    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:10.849748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.849616    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" event={"ID":"3a9441a1-01c5-4113-a10c-59d12ae83b81","Type":"ContainerStarted","Data":"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"}
Oct 31 03:23:10.849748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.849680    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" event={"ID":"3a9441a1-01c5-4113-a10c-59d12ae83b81","Type":"ContainerStarted","Data":"9570c058693bbe54291ec074b1d30a83a49db5bc9bd9b7e6c567220ea2acf2c1"}
Oct 31 03:23:10.849748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.849713    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:10.964135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:10.964074    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" podStartSLOduration=1.9640556359999999 podStartE2EDuration="1.964055636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:10.959390111 +0000 UTC m=+940.973951820" watchObservedRunningTime="2026-10-31 03:23:10.964055636 +0000 UTC m=+940.978617343"
Oct 31 03:23:11.517558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:11.516930    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:11.517558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:11.516992    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:11.549366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:11.549333    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:11.549798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:11.549386    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:12.517562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:12.515426    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:12.517562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:12.515482    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:12.552095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:12.552054    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:12.552661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:12.552131    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:12.978089 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:23:12.978052    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:23:13.538553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:13.518710    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:13.538553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:13.536635    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:13.553415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:13.553002    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:13.553415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:13.553089    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:14.515555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:14.514960    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:14.515555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:14.515029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:14.549836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:14.549803    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:14.550023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:14.549871    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:15.517376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:15.517339    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:15.518014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:15.517978    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:15.549814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:15.549779    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:15.550084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:15.550052    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:16.520345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:16.516915    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:23:16.520345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:16.516976    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:23:16.549693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:16.549575    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:23:16.549693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:16.549657    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 03:23:18.550114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.550071    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:18.550565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.550131    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:18.556667 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.556642    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:18.556771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.556688    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:18.656658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.656627    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:18.657127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.657105    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:18.658151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:18.658129    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:23:19.557287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:19.557248    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:19.557806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:19.557305    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:20.601557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:20.599628    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h2zq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:20.601557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:20.599767    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:21.500364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:21.500252    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:21.501166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:21.501142    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:21.501901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:21.501875    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:23:21.862968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:21.862858    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:23:22.501943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.501902    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:23:22.507443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.507418    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.555974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.555945    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7778d0-openshift-default-dockercfg-c48rb\""
Oct 31 03:23:22.591951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.591916    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.591965    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592082    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjmg\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592193    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592241    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592330    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592378    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592466    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.592514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.592501    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.605669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.605633    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:23:22.693269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693232    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693275    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693375    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693438    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693489    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693573    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693627    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693670    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693739    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjmg\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693761    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.693917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.693823    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.694082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.694001    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.694146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.694073    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.694388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.694329    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.696080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.696058    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.696411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.696395    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.762430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.762354    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjmg\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.762430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.762363    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:22.821960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:22.821922    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:23.300391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.300363    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:23:23.302913 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:23:23.302882    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999063db_d9e8_4bdf_a237_ef3e008ad2db.slice/crio-89e5804d1108a1cb8fe46790fb013763c2fe3899cab8a22edcbd903e230ca63a WatchSource:0}: Error finding container 89e5804d1108a1cb8fe46790fb013763c2fe3899cab8a22edcbd903e230ca63a: Status 404 returned error can't find the container with id 89e5804d1108a1cb8fe46790fb013763c2fe3899cab8a22edcbd903e230ca63a
Oct 31 03:23:23.305364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.305334    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:23.305436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.305401    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:23.305436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.305430    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:23.345772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.345742    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:23:23.351163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.351144    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.392189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.391755    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512384    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmwc\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512487    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512580    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512643    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512682    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512732    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512798    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.512861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512837    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.513448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.512872    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613484    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613592    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613624    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613671    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613718    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613743    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.613780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613768    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613850    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmwc\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.613923    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.614277    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.614295    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.614344    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.614582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.614515    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.615497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.615471    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.617166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.617138    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.617380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.617357    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.668710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.668670    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.669102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.669006    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmwc\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-rpg88\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:23.672237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:23.672211    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:24.018469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.018424    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" event={"ID":"999063db-d9e8-4bdf-a237-ef3e008ad2db","Type":"ContainerStarted","Data":"fa11c25d02db46e6b969fb1bd6c8b6dca15fa9dff926aa38dfbaa916eb5d0594"}
Oct 31 03:23:24.018692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.018565    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" event={"ID":"999063db-d9e8-4bdf-a237-ef3e008ad2db","Type":"ContainerStarted","Data":"89e5804d1108a1cb8fe46790fb013763c2fe3899cab8a22edcbd903e230ca63a"}
Oct 31 03:23:24.125417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.125388    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:23:24.133228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.133189    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:24.133337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.133267    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:24.133337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.133305    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:24.447782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.447729    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podStartSLOduration=2.447711047 podStartE2EDuration="2.447711047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:24.127756138 +0000 UTC m=+954.142317858" watchObservedRunningTime="2026-10-31 03:23:24.447711047 +0000 UTC m=+954.462272756"
Oct 31 03:23:24.448307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.447900    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:24.449347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.449307    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:23:24.476653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.476614    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:23:24.511567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.490819    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.534006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.533957    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:23:24.635260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.635217    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.635513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.635478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpn9g\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.635658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.635517    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.635658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.635596    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.737521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.737424    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpn9g\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.737521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.737476    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.737521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.737516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.737884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.737604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.738497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.738464    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.738803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.738784    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.740258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.740229    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.800757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.800721    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpn9g\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\") pod \"payload-processing-external-model-55bb6bdfd4-l5dgm\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.814462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.814436    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:24.825392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.825366    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:24.829038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.828101    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:24.829038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:24.828153    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:25.041999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.041896    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" event={"ID":"31e80dea-abb0-403f-b5ab-1ccfb17d811e","Type":"ContainerStarted","Data":"c7e19798a61c179c4ac40c77dc797645e923109698d14c0427700753818b6c70"}
Oct 31 03:23:25.041999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.041964    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" event={"ID":"31e80dea-abb0-403f-b5ab-1ccfb17d811e","Type":"ContainerStarted","Data":"6a7c0297b4a8e02d9ec9da8ef4949c4187b439c0eed9dc1b14e2ace2694051be"}
Oct 31 03:23:25.152855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.152768    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podStartSLOduration=2.152753708 podStartE2EDuration="2.152753708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:25.151578177 +0000 UTC m=+955.166139885" watchObservedRunningTime="2026-10-31 03:23:25.152753708 +0000 UTC m=+955.167315416"
Oct 31 03:23:25.262037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.262011    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:23:25.263820 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:23:25.263789    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47be940_723c_4b8f_b68c_4ec719123978.slice/crio-c846e20b36c3078cf5faeda3cee25f16487dcbb78645c622a463df3b8e820257 WatchSource:0}: Error finding container c846e20b36c3078cf5faeda3cee25f16487dcbb78645c622a463df3b8e820257: Status 404 returned error can't find the container with id c846e20b36c3078cf5faeda3cee25f16487dcbb78645c622a463df3b8e820257
Oct 31 03:23:25.672491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.672402    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:25.674149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.674116    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:25.674294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.674156    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:25.825006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.824951    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:25.825225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:25.825017    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:26.050729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.050695    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" event={"ID":"f47be940-723c-4b8f-b68c-4ec719123978","Type":"ContainerStarted","Data":"13d1370a780e33163955a617d92c2019705dcd1e900bf1ce9f64210fbb7e48b2"}
Oct 31 03:23:26.050921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.050745    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" event={"ID":"f47be940-723c-4b8f-b68c-4ec719123978","Type":"ContainerStarted","Data":"c846e20b36c3078cf5faeda3cee25f16487dcbb78645c622a463df3b8e820257"}
Oct 31 03:23:26.051189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.051156    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:23:26.160367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.160311    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podStartSLOduration=2.160297631 podStartE2EDuration="2.160297631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:26.156347795 +0000 UTC m=+956.170909502" watchObservedRunningTime="2026-10-31 03:23:26.160297631 +0000 UTC m=+956.174859339"
Oct 31 03:23:26.673479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.673444    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:26.673906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.673498    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:26.826555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.825959    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:26.826555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:26.826019    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:27.673382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:27.673340    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:27.673632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:27.673399    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:27.828579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:27.825987    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:27.828579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:27.826053    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:28.673161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:28.673114    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:28.673344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:28.673184    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:28.827122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:28.827088    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:28.827773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:28.827726    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:29.448411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.448359    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing" containerID="cri-o://2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5" gracePeriod=25
Oct 31 03:23:29.673890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.673417    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:29.673890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.673478    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:29.823551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.822806    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 03:23:29.823551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.822864    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 03:23:29.829571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.826168    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:29.944856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.944823    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\") pod \"3a9441a1-01c5-4113-a10c-59d12ae83b81\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") "
Oct 31 03:23:29.945029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.945015    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\") pod \"3a9441a1-01c5-4113-a10c-59d12ae83b81\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") "
Oct 31 03:23:29.945129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.945095    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\") pod \"3a9441a1-01c5-4113-a10c-59d12ae83b81\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") "
Oct 31 03:23:29.945129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.945123    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\") pod \"3a9441a1-01c5-4113-a10c-59d12ae83b81\" (UID: \"3a9441a1-01c5-4113-a10c-59d12ae83b81\") "
Oct 31 03:23:29.945256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.945168    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume" pod "3a9441a1-01c5-4113-a10c-59d12ae83b81" (UID: "3a9441a1-01c5-4113-a10c-59d12ae83b81"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:23:29.945620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.945585    5831 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-routing-overlay-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:23:29.946063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.946036    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume" pod "3a9441a1-01c5-4113-a10c-59d12ae83b81" (UID: "3a9441a1-01c5-4113-a10c-59d12ae83b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:23:29.948720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.948671    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m" pod "3a9441a1-01c5-4113-a10c-59d12ae83b81" (UID: "3a9441a1-01c5-4113-a10c-59d12ae83b81"). InnerVolumeSpecName "kube-api-access-rxv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:29.948853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:29.948753    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials" pod "3a9441a1-01c5-4113-a10c-59d12ae83b81" (UID: "3a9441a1-01c5-4113-a10c-59d12ae83b81"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:30.051856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.050458    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv2m\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-kube-api-access-rxv2m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:23:30.051856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.051739    5831 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a9441a1-01c5-4113-a10c-59d12ae83b81-provider-credentials\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:23:30.051856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.051771    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9441a1-01c5-4113-a10c-59d12ae83b81-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:23:30.124700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.124610    5831 generic.go:356] "Generic (PLEG): container finished" podID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerID="2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5" exitCode=0
Oct 31 03:23:30.124957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.124683    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"
Oct 31 03:23:30.125037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.124683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" event={"ID":"3a9441a1-01c5-4113-a10c-59d12ae83b81","Type":"ContainerDied","Data":"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"}
Oct 31 03:23:30.125096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.125072    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm" event={"ID":"3a9441a1-01c5-4113-a10c-59d12ae83b81","Type":"ContainerDied","Data":"9570c058693bbe54291ec074b1d30a83a49db5bc9bd9b7e6c567220ea2acf2c1"}
Oct 31 03:23:30.125152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.125103    5831 scope.go:122] "RemoveContainer" containerID="2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"
Oct 31 03:23:30.141345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.141330    5831 scope.go:122] "RemoveContainer" containerID="2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"
Oct 31 03:23:30.142753 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:23:30.142685    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5\": container with ID starting with 2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5 not found: ID does not exist" containerID="2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"
Oct 31 03:23:30.142753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.142717    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5"} err="failed to get container status \"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5\": rpc error: code = NotFound desc = could not find container \"2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5\": container with ID starting with 2c9db8633412ed327b8868905d41cd9aa90617f605804f9992b86d9dd025f4e5 not found: ID does not exist"
Oct 31 03:23:30.313614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.312584    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:30.327295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.327264    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54b7cc8547-64sbm"]
Oct 31 03:23:30.672911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.672827    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:30.672911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.672880    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:30.827732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.827701    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:30.828129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.828104    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:30.828852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:30.828833    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:23:31.672954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:31.672919    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:23:31.673349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:31.672970    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:23:31.997357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:31.997317    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:23:32.136238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:32.136198    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" path="/var/lib/kubelet/pods/3a9441a1-01c5-4113-a10c-59d12ae83b81/volumes"
Oct 31 03:23:32.963236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:32.963194    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:32.964106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:32.963855    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:32.964899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:32.964879    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:23:33.873358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.873308    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:23:33.874214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.874187    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing"
Oct 31 03:23:33.874214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.874215    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing"
Oct 31 03:23:33.874431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.874408    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9441a1-01c5-4113-a10c-59d12ae83b81" containerName="payload-processing"
Oct 31 03:23:33.878940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.878903    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.935654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.935612    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.935988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.935958    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.936227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.936206    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.936397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.936378    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.936592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.936515    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.936763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.936745    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.936904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.936886    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.937054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.937036    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds6hj\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.937199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.937183    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:33.971425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:33.967055    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:23:34.038643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.038591    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.039186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.039157    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds6hj\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.039696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.039669    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.040061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.040036    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.040502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.040478    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.040622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.040565    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.040693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.040638    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.040966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.040938    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.041104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.041085    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.041689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.041665    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.041795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.041710    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.041795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.041769    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.042513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.042481    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.043225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.042870    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.045429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.045399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.046344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.046298    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.093516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.093484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds6hj\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.102691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.097815    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:34.200995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:34.200823    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:35.033555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:35.032819    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:23:35.137577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:35.132348    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:35.137577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:35.132442    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:35.137577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:35.132486    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:23:35.210508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:35.210472    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" event={"ID":"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1","Type":"ContainerStarted","Data":"49abf93037d755d6e9d354d9b1b5645d4d0d23d2b10c154e365f6432e3ee34e5"}
Oct 31 03:23:36.354986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:36.354940    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" event={"ID":"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1","Type":"ContainerStarted","Data":"358c9bd42d1170edc9364a2af12b5f1a48bda91e5111faf4c7b7b740939d5c60"}
Oct 31 03:23:36.581969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:36.581907    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podStartSLOduration=3.581888117 podStartE2EDuration="3.581888117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:36.525239529 +0000 UTC m=+966.539801263" watchObservedRunningTime="2026-10-31 03:23:36.581888117 +0000 UTC m=+966.596449826"
Oct 31 03:23:37.203795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:37.203752    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:37.205414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:37.205388    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:37.205543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:37.205439    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:38.075107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:38.075075    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:23:38.202564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:38.201962    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:38.202564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:38.202029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:39.204947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:39.204906    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:39.205767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:39.205735    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:40.204006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:40.203968    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:40.204230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:40.204021    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:41.203969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:41.203930    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:41.204409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:41.204000    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:42.201660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:42.201617    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:23:42.201882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:42.201684    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 31 03:23:43.079332 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:23:43.079270    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:23:43.458009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:43.457974    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:43.458382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:43.458348    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:43.459948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:43.459918    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:23:48.072893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:48.072852    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:23:58.073241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:23:58.073202    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:24:00.926677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:00.926639    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:24:08.073785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:08.073744    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:24:12.144634 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.144601    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.144634 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.144636    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.145218 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.144715    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:12.644695202 +0000 UTC m=+1002.659256907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.648768 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.648740    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.648768 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.648771    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.648980 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.648831    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:13.64881487 +0000 UTC m=+1003.663376559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:12.978380 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:12.978342    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:24:13.666558 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:13.663857    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:13.666558 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:13.663896    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:13.666558 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:13.663956    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:15.663936727 +0000 UTC m=+1005.678498416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:15.725739 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:15.725704    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:15.725739 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:15.725734    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:15.726255 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:15.725806    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:19.725788509 +0000 UTC m=+1009.740350200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:17.626451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.626420    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:24:17.628032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.627993    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:24:17.655257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.655225    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:24:17.660926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.660901    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.709072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.709042    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9bdee0111b09\""
Oct 31 03:24:17.709072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.709073    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a1ad4b4e573c\""
Oct 31 03:24:17.731004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.730973    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:24:17.749377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.749252    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh48w\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.751933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.751898    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.752128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.752045    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.752230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.752204    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.853652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.853615    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh48w\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.853839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.853677    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.853839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.853738    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.853839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.853823    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.854575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.854524    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.854719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.854693    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.856609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.856581    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.906539 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.906431    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh48w\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\") pod \"payload-processing-external-model-5dfbf6cc9d-sxpft\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") " pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:17.975908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:17.975877    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:18.350250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:18.350220    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:24:18.351759 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:24:18.351733    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2b1200_ad45_462d_ab2f_c1b9a75d1c64.slice/crio-4fe1e7e97e09ca7030fea6a97ded8e73361af00d1f26ac252be6c1b82c206147 WatchSource:0}: Error finding container 4fe1e7e97e09ca7030fea6a97ded8e73361af00d1f26ac252be6c1b82c206147: Status 404 returned error can't find the container with id 4fe1e7e97e09ca7030fea6a97ded8e73361af00d1f26ac252be6c1b82c206147
Oct 31 03:24:18.782578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:18.782540    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" event={"ID":"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64","Type":"ContainerStarted","Data":"2f60136f95174d1f779498f335379e2253b2bf016fc931b5fa2616b1c251b8ab"}
Oct 31 03:24:18.782997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:18.782606    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" event={"ID":"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64","Type":"ContainerStarted","Data":"4fe1e7e97e09ca7030fea6a97ded8e73361af00d1f26ac252be6c1b82c206147"}
Oct 31 03:24:18.782997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:18.782657    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:18.892424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:18.892368    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" podStartSLOduration=1.8923527629999999 podStartE2EDuration="1.892352763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:18.889422854 +0000 UTC m=+1008.903984575" watchObservedRunningTime="2026-10-31 03:24:18.892352763 +0000 UTC m=+1008.906914471"
Oct 31 03:24:19.782723 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:19.782688    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:19.782723 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:19.782723    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:19.783230 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:19.782780    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:27.782761513 +0000 UTC m=+1017.797323199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:22.104614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:22.104581    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:24:22.627554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:22.626738    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing" containerID="cri-o://13d1370a780e33163955a617d92c2019705dcd1e900bf1ce9f64210fbb7e48b2" gracePeriod=25
Oct 31 03:24:22.842552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:22.835876    5831 generic.go:356] "Generic (PLEG): container finished" podID="f47be940-723c-4b8f-b68c-4ec719123978" containerID="13d1370a780e33163955a617d92c2019705dcd1e900bf1ce9f64210fbb7e48b2" exitCode=0
Oct 31 03:24:22.842552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:22.835922    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" event={"ID":"f47be940-723c-4b8f-b68c-4ec719123978","Type":"ContainerDied","Data":"13d1370a780e33163955a617d92c2019705dcd1e900bf1ce9f64210fbb7e48b2"}
Oct 31 03:24:23.622218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.621817    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:24:23.705591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.705318    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\") pod \"f47be940-723c-4b8f-b68c-4ec719123978\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") "
Oct 31 03:24:23.705591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.705496    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\") pod \"f47be940-723c-4b8f-b68c-4ec719123978\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") "
Oct 31 03:24:23.709553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.706023    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume" pod "f47be940-723c-4b8f-b68c-4ec719123978" (UID: "f47be940-723c-4b8f-b68c-4ec719123978"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:24:23.709553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.706801    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume" pod "f47be940-723c-4b8f-b68c-4ec719123978" (UID: "f47be940-723c-4b8f-b68c-4ec719123978"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:24:23.709553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.706199    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\") pod \"f47be940-723c-4b8f-b68c-4ec719123978\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") "
Oct 31 03:24:23.709553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.707475    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\") pod \"f47be940-723c-4b8f-b68c-4ec719123978\" (UID: \"f47be940-723c-4b8f-b68c-4ec719123978\") "
Oct 31 03:24:23.709553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.709123    5831 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-routing-overlay-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:23.709963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.709701    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47be940-723c-4b8f-b68c-4ec719123978-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:23.721735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.710683    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials" pod "f47be940-723c-4b8f-b68c-4ec719123978" (UID: "f47be940-723c-4b8f-b68c-4ec719123978"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:23.736875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.733691    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g" pod "f47be940-723c-4b8f-b68c-4ec719123978" (UID: "f47be940-723c-4b8f-b68c-4ec719123978"). InnerVolumeSpecName "kube-api-access-gpn9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:23.820782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.820057    5831 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-provider-credentials\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:23.820782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.820100    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpn9g\" (UniqueName: \"kubernetes.io/projected/f47be940-723c-4b8f-b68c-4ec719123978-kube-api-access-gpn9g\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:23.897555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.897202    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm" event={"ID":"f47be940-723c-4b8f-b68c-4ec719123978","Type":"ContainerDied","Data":"c846e20b36c3078cf5faeda3cee25f16487dcbb78645c622a463df3b8e820257"}
Oct 31 03:24:23.897555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.897314    5831 scope.go:122] "RemoveContainer" containerID="13d1370a780e33163955a617d92c2019705dcd1e900bf1ce9f64210fbb7e48b2"
Oct 31 03:24:23.897555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:23.897560    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"
Oct 31 03:24:24.219811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:24.219769    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:24:24.292560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:24.273860    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-l5dgm"]
Oct 31 03:24:25.455565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:25.454741    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:24:25.455565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:25.454854    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:24:26.152279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:26.151807    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47be940-723c-4b8f-b68c-4ec719123978" path="/var/lib/kubelet/pods/f47be940-723c-4b8f-b68c-4ec719123978/volumes"
Oct 31 03:24:27.829707 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:27.829672    5831 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-280afa1d2683: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:27.829707 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:27.829703    5831 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv: secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:27.830142 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:27.829782    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials podName:8b235195-3c14-4aea-af46-cd14114ac5e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:24:43.829767575 +0000 UTC m=+1033.844329281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials") pod "payload-processing-external-model-5b7c6c55f6-7m6tv" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5") : secret "agc-cred-280afa1d2683" not found
Oct 31 03:24:29.796374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:29.796345    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:24:29.954523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:29.954493    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:24:30.350090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:30.350061    5831 scope.go:122] "RemoveContainer" containerID="2bc7b55a11abecbdcdc1df920cbaa7caf42fd7d14236bea2fa2c1de02de146dd"
Oct 31 03:24:30.360435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:30.360411    5831 scope.go:122] "RemoveContainer" containerID="45baf147f06e41ba02f741c7600cf369f0e52380a9ce08a8447ad74fd53a287b"
Oct 31 03:24:30.369801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:30.369782    5831 scope.go:122] "RemoveContainer" containerID="6d8dd39f8e05f80f67031272d206ab018782e476237584a5078b228a6619f235"
Oct 31 03:24:30.382776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:30.382720    5831 scope.go:122] "RemoveContainer" containerID="cb178313a9173960be2e8aefb2e384237e12bcd6974dd5e8a8f514b4bddab941"
Oct 31 03:24:34.955621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:34.955580    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" podUID="8b235195-3c14-4aea-af46-cd14114ac5e5" containerName="payload-processing" containerID="cri-o://13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5" gracePeriod=25
Oct 31 03:24:35.218507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.218438    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:24:35.319854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.319816    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\") pod \"8b235195-3c14-4aea-af46-cd14114ac5e5\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") "
Oct 31 03:24:35.320049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.319867    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\") pod \"8b235195-3c14-4aea-af46-cd14114ac5e5\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") "
Oct 31 03:24:35.320049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.319894    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\") pod \"8b235195-3c14-4aea-af46-cd14114ac5e5\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") "
Oct 31 03:24:35.320049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.319957    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\") pod \"8b235195-3c14-4aea-af46-cd14114ac5e5\" (UID: \"8b235195-3c14-4aea-af46-cd14114ac5e5\") "
Oct 31 03:24:35.320292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.320264    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume" pod "8b235195-3c14-4aea-af46-cd14114ac5e5" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:24:35.320369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.320310    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume" pod "8b235195-3c14-4aea-af46-cd14114ac5e5" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:24:35.320424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.320407    5831 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-routing-overlay-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:35.320460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.320428    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b235195-3c14-4aea-af46-cd14114ac5e5-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:35.322382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.322352    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg" pod "8b235195-3c14-4aea-af46-cd14114ac5e5" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5"). InnerVolumeSpecName "kube-api-access-26fxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:35.322496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.322408    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials" pod "8b235195-3c14-4aea-af46-cd14114ac5e5" (UID: "8b235195-3c14-4aea-af46-cd14114ac5e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:35.386118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.386082    5831 generic.go:356] "Generic (PLEG): container finished" podID="8b235195-3c14-4aea-af46-cd14114ac5e5" containerID="13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5" exitCode=0
Oct 31 03:24:35.386289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.386132    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"
Oct 31 03:24:35.386289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.386169    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" event={"ID":"8b235195-3c14-4aea-af46-cd14114ac5e5","Type":"ContainerDied","Data":"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"}
Oct 31 03:24:35.386289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.386229    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv" event={"ID":"8b235195-3c14-4aea-af46-cd14114ac5e5","Type":"ContainerDied","Data":"dd55f20563111969f733e5d1bf48c28031e5f375fb86641a8c44df6f1c85345a"}
Oct 31 03:24:35.386289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.386255    5831 scope.go:122] "RemoveContainer" containerID="13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"
Oct 31 03:24:35.398093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.398073    5831 scope.go:122] "RemoveContainer" containerID="13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"
Oct 31 03:24:35.398387 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:35.398369    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5\": container with ID starting with 13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5 not found: ID does not exist" containerID="13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"
Oct 31 03:24:35.398440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.398397    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5"} err="failed to get container status \"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5\": rpc error: code = NotFound desc = could not find container \"13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5\": container with ID starting with 13977ae4c391aa867087aee4821b457411ab9b1d9d591539cfd56dc1fd5b7ac5 not found: ID does not exist"
Oct 31 03:24:35.421924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.421896    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fxg\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-kube-api-access-26fxg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:35.421924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.421927    5831 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b235195-3c14-4aea-af46-cd14114ac5e5-provider-credentials\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:24:35.541164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.541084    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:24:35.546500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:35.546475    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b7c6c55f6-7m6tv"]
Oct 31 03:24:36.135259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:36.135227    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b235195-3c14-4aea-af46-cd14114ac5e5" path="/var/lib/kubelet/pods/8b235195-3c14-4aea-af46-cd14114ac5e5/volumes"
Oct 31 03:24:43.185556 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:24:43.185466    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:24:49.438281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.435038    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:24:49.439087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439060    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing"
Oct 31 03:24:49.439248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439234    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing"
Oct 31 03:24:49.439370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439359    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8b235195-3c14-4aea-af46-cd14114ac5e5" containerName="payload-processing"
Oct 31 03:24:49.439448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439437    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b235195-3c14-4aea-af46-cd14114ac5e5" containerName="payload-processing"
Oct 31 03:24:49.439737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439720    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b235195-3c14-4aea-af46-cd14114ac5e5" containerName="payload-processing"
Oct 31 03:24:49.439895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.439882    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47be940-723c-4b8f-b68c-4ec719123978" containerName="payload-processing"
Oct 31 03:24:49.448150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.448125    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.473376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.473347    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:24:49.477357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.477338    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.533838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.533803    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.533838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.533841    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.533874    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.533975    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534012    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534035    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534137    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534154    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534175    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534196    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534295    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534328    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6df8f\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534361    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534434    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534456    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534477    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdq2m\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.534605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534568    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.534668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.534655    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.550687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.550659    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:24:49.577458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.577427    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:24:49.635957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.635920    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.635964    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636007    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6df8f\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636042    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636080    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636099    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636138    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdq2m\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636182    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636248    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636277    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636305    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636347    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636347    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636388    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636413    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636448    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.636492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636490    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636511    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636702    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636778    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636816    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636833    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.636449    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.637217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.637075    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.637673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.637326    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.637333    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.637673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.637552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.638981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.638961    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.639328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.639304    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.639492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.639469    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.639697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.639671    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.698118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.691612    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdq2m\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.698118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.692008    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.698118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.692781    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.698118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.692932    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6df8f\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:49.763997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.763964    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:49.791397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:49.791161    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:50.197080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.197049    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:24:50.200286 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:24:50.200171    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5906ea5c_3b72_4661_b0a0_663d0fa093bb.slice/crio-77de1f685b7a4a11296e9785904b0b78c77f4c2a519982be44f9361ef4afdea5 WatchSource:0}: Error finding container 77de1f685b7a4a11296e9785904b0b78c77f4c2a519982be44f9361ef4afdea5: Status 404 returned error can't find the container with id 77de1f685b7a4a11296e9785904b0b78c77f4c2a519982be44f9361ef4afdea5
Oct 31 03:24:50.202496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.202473    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:24:50.202904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.202870    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.202990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.202935    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.202990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.202962    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.205345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.205315    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.205449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.205393    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.205449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.205438    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:24:50.577961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.577862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" event={"ID":"5906ea5c-3b72-4661-b0a0-663d0fa093bb","Type":"ContainerStarted","Data":"2badf9ddb7c0ca47af126fee754bf9a0b4c95b79fad35f84c7b29a05d4589cc4"}
Oct 31 03:24:50.577961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.577933    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" event={"ID":"5906ea5c-3b72-4661-b0a0-663d0fa093bb","Type":"ContainerStarted","Data":"77de1f685b7a4a11296e9785904b0b78c77f4c2a519982be44f9361ef4afdea5"}
Oct 31 03:24:50.579979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.579950    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" event={"ID":"91d64a96-6a85-490a-9f7e-377d9ab3ca18","Type":"ContainerStarted","Data":"1cc98893d83b376e0869c44dde588b653f62484e465b3f4065c5e43216e5b188"}
Oct 31 03:24:50.580113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.580004    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" event={"ID":"91d64a96-6a85-490a-9f7e-377d9ab3ca18","Type":"ContainerStarted","Data":"96d0399937c8e710f73df4eb00b9d4dc56c49c3d528ee564509f3aa74365472b"}
Oct 31 03:24:50.699143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.699085    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podStartSLOduration=1.699065687 podStartE2EDuration="1.699065687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:50.689728443 +0000 UTC m=+1040.704290153" watchObservedRunningTime="2026-10-31 03:24:50.699065687 +0000 UTC m=+1040.713627397"
Oct 31 03:24:50.764973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.764943    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:50.791561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.791518    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:50.799654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:50.799597    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podStartSLOduration=1.799579924 podStartE2EDuration="1.799579924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:50.798633111 +0000 UTC m=+1040.813194818" watchObservedRunningTime="2026-10-31 03:24:50.799579924 +0000 UTC m=+1040.814141638"
Oct 31 03:24:51.766294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:51.766264    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:51.766737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:51.766310    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:51.798557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:51.794682    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:51.798557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:51.794750    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:52.765022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:52.764982    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:52.765210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:52.765032    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:52.799556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:52.795941    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:52.799556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:52.796004    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:53.764643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:53.764611    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:53.764907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:53.764881    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:53.793016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:53.792974    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:53.794047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:53.794009    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:53.849250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:53.849212    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:24:54.782702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:54.782668    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:54.783015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:54.782982    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:54.801320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:54.801284    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:54.801695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:54.801663    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:55.770409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:55.769939    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:55.770409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:55.770001    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:55.828980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:55.828944    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:55.829300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:55.829271    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:56.764889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:56.764852    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:56.765179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:56.765149    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:56.795710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:56.795671    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:56.796266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:56.796237    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:57.765169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:57.765131    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:24:57.765399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:57.765212    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 03:24:57.791608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:57.791563    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:24:57.791815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:57.791628    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:24:58.769123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.769088    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:58.769585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.769571    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:58.770177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.770159    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:24:58.796012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.795988    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:58.796400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.796376    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:24:58.797225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:24:58.797208    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:25:12.977864 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:25:12.977826    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:25:15.633648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.633599    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:25:15.637494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.637473    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.730613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.730576    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.730905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.730880    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fbg\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.730938    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731025    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731045    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731156    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731217    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731419    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.731572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.731478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.735589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.735568    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:25:15.832770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832733    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.832972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.832972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832817    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.832972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832844    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.832972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832895    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.832987    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833043    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833116    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833163    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fbg\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833193    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833571    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833596    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.833934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.833913    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.835640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.835616    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.835748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.835682    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.886781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.886700    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fbg\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.887118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.887097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:15.951844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:15.951812    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:16.379267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.379237    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:25:16.379690 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:16.379663    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c132719_ed85_457c_aeb8_2e5a2d49c15e.slice/crio-36d03662f0cc68e3074260dd7711fa6d0213ecda9b72f2d2f80f4a1edf4514cb WatchSource:0}: Error finding container 36d03662f0cc68e3074260dd7711fa6d0213ecda9b72f2d2f80f4a1edf4514cb: Status 404 returned error can't find the container with id 36d03662f0cc68e3074260dd7711fa6d0213ecda9b72f2d2f80f4a1edf4514cb
Oct 31 03:25:16.382053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.382025    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:16.382121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.382107    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:16.382158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.382137    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:16.708483 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.708448    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"]
Oct 31 03:25:16.713053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.713029    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.764501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.764473    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:25:16.764700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.764547    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:25:16.775596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.775565    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"]
Oct 31 03:25:16.845292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.845254    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjhx\" (UniqueName: \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.845477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.845424    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.845651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.845564    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.864705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.864678    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:25:16.864705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.864698    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:25:16.948353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.948294    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjhx\" (UniqueName: \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.948582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.948431    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.948582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.948559    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.949696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.949666    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.951372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.951329    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:16.968923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.968841    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" event={"ID":"9c132719-ed85-457c-aeb8-2e5a2d49c15e","Type":"ContainerStarted","Data":"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"}
Oct 31 03:25:16.968923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:16.968898    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" event={"ID":"9c132719-ed85-457c-aeb8-2e5a2d49c15e","Type":"ContainerStarted","Data":"36d03662f0cc68e3074260dd7711fa6d0213ecda9b72f2d2f80f4a1edf4514cb"}
Oct 31 03:25:17.007555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.007428    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjhx\" (UniqueName: \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\") pod \"maas-api-revoke-sub-8fb5b28f-4mvcl\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:17.027586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.027558    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:17.435616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.435546    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podStartSLOduration=2.435507898 podStartE2EDuration="2.435507898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:17.100466378 +0000 UTC m=+1067.115028087" watchObservedRunningTime="2026-10-31 03:25:17.435507898 +0000 UTC m=+1067.450069607"
Oct 31 03:25:17.436886 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:17.436862    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38315051_94bb_49e4_a822_2e119454c0a6.slice/crio-e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7 WatchSource:0}: Error finding container e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7: Status 404 returned error can't find the container with id e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7
Oct 31 03:25:17.437763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.437741    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"]
Oct 31 03:25:17.952469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.952433    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:17.954409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.954382    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:17.954551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.954433    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:17.978210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.978178    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl" event={"ID":"38315051-94bb-49e4-a822-2e119454c0a6","Type":"ContainerStarted","Data":"2b27cf4da9e1874029502383bfe10060b0bbc320905e22b7964297a4ad80a820"}
Oct 31 03:25:17.978376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:17.978230    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl" event={"ID":"38315051-94bb-49e4-a822-2e119454c0a6","Type":"ContainerStarted","Data":"e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7"}
Oct 31 03:25:18.100613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:18.097724    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl" podStartSLOduration=2.097703858 podStartE2EDuration="2.097703858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:18.094898185 +0000 UTC m=+1068.109459895" watchObservedRunningTime="2026-10-31 03:25:18.097703858 +0000 UTC m=+1068.112265588"
Oct 31 03:25:18.952298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:18.952259    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:18.952488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:18.952326    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:19.008635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:19.008578    5831 generic.go:356] "Generic (PLEG): container finished" podID="38315051-94bb-49e4-a822-2e119454c0a6" containerID="2b27cf4da9e1874029502383bfe10060b0bbc320905e22b7964297a4ad80a820" exitCode=0
Oct 31 03:25:19.009826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:19.009804    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl" event={"ID":"38315051-94bb-49e4-a822-2e119454c0a6","Type":"ContainerDied","Data":"2b27cf4da9e1874029502383bfe10060b0bbc320905e22b7964297a4ad80a820"}
Oct 31 03:25:19.952421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:19.952385    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:19.952653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:19.952452    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:20.230651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.230568    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:20.319041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.319007    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\" (UniqueName: \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\") pod \"38315051-94bb-49e4-a822-2e119454c0a6\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") "
Oct 31 03:25:20.319249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.319146    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\") pod \"38315051-94bb-49e4-a822-2e119454c0a6\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") "
Oct 31 03:25:20.319293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.319251    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\") pod \"38315051-94bb-49e4-a822-2e119454c0a6\" (UID: \"38315051-94bb-49e4-a822-2e119454c0a6\") "
Oct 31 03:25:20.319594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.319566    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca" pod "38315051-94bb-49e4-a822-2e119454c0a6" (UID: "38315051-94bb-49e4-a822-2e119454c0a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:25:20.319828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.319804    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:20.321412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.321384    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx" pod "38315051-94bb-49e4-a822-2e119454c0a6" (UID: "38315051-94bb-49e4-a822-2e119454c0a6"). InnerVolumeSpecName "kube-api-access-fmjhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:20.321512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.321495    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca" pod "38315051-94bb-49e4-a822-2e119454c0a6" (UID: "38315051-94bb-49e4-a822-2e119454c0a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:20.421956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.421917    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38315051-94bb-49e4-a822-2e119454c0a6-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:20.421956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.421957    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmjhx\" (UniqueName: \"kubernetes.io/projected/38315051-94bb-49e4-a822-2e119454c0a6-kube-api-access-fmjhx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:20.952702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.952671    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:20.953112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:20.952735    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:21.028035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:21.027999    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl" event={"ID":"38315051-94bb-49e4-a822-2e119454c0a6","Type":"ContainerDied","Data":"e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7"}
Oct 31 03:25:21.028208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:21.028066    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a19b109bfc68539b356dada765e6b56826001d83221a5fd594c3e6df359ef7"
Oct 31 03:25:21.028208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:21.028007    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"
Oct 31 03:25:21.952131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:21.952099    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:21.952310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:21.952152    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:22.865130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.865095    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:25:22.865709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.865695    5831 cpu_manager.go:414] "containerMap: removing container" podUID="38315051-94bb-49e4-a822-2e119454c0a6" containerName="revoke-keys"
Oct 31 03:25:22.865760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.865711    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38315051-94bb-49e4-a822-2e119454c0a6" containerName="revoke-keys"
Oct 31 03:25:22.865820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.865810    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="38315051-94bb-49e4-a822-2e119454c0a6" containerName="revoke-keys"
Oct 31 03:25:22.869646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.869617    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:22.953094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.953057    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:25:22.953298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.953126    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:25:22.968807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:22.968767    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:25:23.056883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.056851    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxsl\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.056890    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.056926    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057034    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057086    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057171    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057203    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057319    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.057391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.057378    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159071    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159160    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159225    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159259    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159352    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159396    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159444    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159472    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxsl\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159506    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159597    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.159659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159640    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.160021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159890    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.160021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.159961    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.160741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.160704    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.162256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.162229    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.162355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.162301    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.216142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.216113    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.216394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.216370    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxsl\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.484840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.484804    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:23.935716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.935685    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:25:23.936136 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:23.935947    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafbce92_55cb_4bba_b666_504112c0cc11.slice/crio-bd4514fff83a69d9887fa189d89d720e3d9c051a0d03ef9d3c8360720f7d815a WatchSource:0}: Error finding container bd4514fff83a69d9887fa189d89d720e3d9c051a0d03ef9d3c8360720f7d815a: Status 404 returned error can't find the container with id bd4514fff83a69d9887fa189d89d720e3d9c051a0d03ef9d3c8360720f7d815a
Oct 31 03:25:23.938696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.938656    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:23.938784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.938768    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:23.938826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.938814    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:23.958418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.958389    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:23.959191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.958806    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:23.959838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:23.959816    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:25:24.070809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:24.070714    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" event={"ID":"eafbce92-55cb-4bba-b666-504112c0cc11","Type":"ContainerStarted","Data":"bc878ce8a8227ce834370a275b8a4ae5250bee5da920695237e4758d54c017e3"}
Oct 31 03:25:24.070809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:24.070777    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" event={"ID":"eafbce92-55cb-4bba-b666-504112c0cc11","Type":"ContainerStarted","Data":"bd4514fff83a69d9887fa189d89d720e3d9c051a0d03ef9d3c8360720f7d815a"}
Oct 31 03:25:24.166275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:24.166236    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:25:24.485939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:24.485902    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:25.493808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:25.493770    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:25.494436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:25.494404    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:26.495556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:26.489934    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:26.495556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:26.489995    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:27.490924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.490889    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:27.491204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.491175    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:27.514156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.514084    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podStartSLOduration=5.514063803 podStartE2EDuration="5.514063803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:24.304863853 +0000 UTC m=+1074.319425586" watchObservedRunningTime="2026-10-31 03:25:27.514063803 +0000 UTC m=+1077.528625515"
Oct 31 03:25:27.517173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.517148    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"]
Oct 31 03:25:27.523824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.523805    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.567658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.567630    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"]
Oct 31 03:25:27.571973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.571942    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.572126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.571992    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94zz\" (UniqueName: \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.572212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.572192    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.608555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.596880    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:25:27.613552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.609836    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:25:27.613552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.609942    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:25:27.613552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.609971    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:25:27.673708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.673665    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.673906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.673715    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94zz\" (UniqueName: \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.673906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.673854    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.674450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.674431    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.676180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.676157    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.728827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.728791    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94zz\" (UniqueName: \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\") pod \"maas-api-revoke-sub-f5516e46-zlm84\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:27.841791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:27.841717    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:28.248595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:28.248564    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"]
Oct 31 03:25:28.249605 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:28.249576    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3935c86a_5f18_4ade_83a0_ccddfe4e088d.slice/crio-944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906 WatchSource:0}: Error finding container 944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906: Status 404 returned error can't find the container with id 944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906
Oct 31 03:25:28.490033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:28.489973    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:28.490236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:28.490039    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:29.215928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:29.215887    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84" event={"ID":"3935c86a-5f18-4ade-83a0-ccddfe4e088d","Type":"ContainerStarted","Data":"eb62b933fc2e65f4bc6cda8c82ba0b2f8b3539c36e74d650ab8f4f7587daa58a"}
Oct 31 03:25:29.216423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:29.215946    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84" event={"ID":"3935c86a-5f18-4ade-83a0-ccddfe4e088d","Type":"ContainerStarted","Data":"944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906"}
Oct 31 03:25:29.323162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:29.323095    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84" podStartSLOduration=2.323074256 podStartE2EDuration="2.323074256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:29.320609938 +0000 UTC m=+1079.335171647" watchObservedRunningTime="2026-10-31 03:25:29.323074256 +0000 UTC m=+1079.337635969"
Oct 31 03:25:29.487998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:29.487903    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:29.488257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:29.488226    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:30.228030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:30.227992    5831 generic.go:356] "Generic (PLEG): container finished" podID="3935c86a-5f18-4ade-83a0-ccddfe4e088d" containerID="eb62b933fc2e65f4bc6cda8c82ba0b2f8b3539c36e74d650ab8f4f7587daa58a" exitCode=0
Oct 31 03:25:30.228566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:30.228038    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84" event={"ID":"3935c86a-5f18-4ade-83a0-ccddfe4e088d","Type":"ContainerDied","Data":"eb62b933fc2e65f4bc6cda8c82ba0b2f8b3539c36e74d650ab8f4f7587daa58a"}
Oct 31 03:25:30.431208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:30.431180    5831 scope.go:122] "RemoveContainer" containerID="53400b32873c5a06c51ceb8d7ddd1d72232630d786cd8960e1194f96160ca5a5"
Oct 31 03:25:30.486210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:30.486119    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:25:30.486210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:30.486186    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:25:31.406671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.406643    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:31.570517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.570432    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\" (UniqueName: \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\") pod \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") "
Oct 31 03:25:31.570695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.570620    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\") pod \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") "
Oct 31 03:25:31.570812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.570796    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\") pod \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\" (UID: \"3935c86a-5f18-4ade-83a0-ccddfe4e088d\") "
Oct 31 03:25:31.571043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.571012    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca" pod "3935c86a-5f18-4ade-83a0-ccddfe4e088d" (UID: "3935c86a-5f18-4ade-83a0-ccddfe4e088d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:25:31.571357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.571325    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:31.572696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.572671    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz" pod "3935c86a-5f18-4ade-83a0-ccddfe4e088d" (UID: "3935c86a-5f18-4ade-83a0-ccddfe4e088d"). InnerVolumeSpecName "kube-api-access-q94zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:31.572976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.572956    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca" pod "3935c86a-5f18-4ade-83a0-ccddfe4e088d" (UID: "3935c86a-5f18-4ade-83a0-ccddfe4e088d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:31.673290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.673248    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3935c86a-5f18-4ade-83a0-ccddfe4e088d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:31.673290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:31.673291    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94zz\" (UniqueName: \"kubernetes.io/projected/3935c86a-5f18-4ade-83a0-ccddfe4e088d-kube-api-access-q94zz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:32.250808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.250779    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"
Oct 31 03:25:32.250981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.250804    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84" event={"ID":"3935c86a-5f18-4ade-83a0-ccddfe4e088d","Type":"ContainerDied","Data":"944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906"}
Oct 31 03:25:32.250981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.250863    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944b6c1501b249c30ca1b9b3abb5a86497bc32a36317f256a42e09f4f6768906"
Oct 31 03:25:32.526723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.526680    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:25:32.527185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.526758    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:25:32.663585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.663549    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:32.664009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.663988    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:32.679571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:32.679502    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:25:37.986793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:37.986758    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:25:37.987354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:37.987339    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3935c86a-5f18-4ade-83a0-ccddfe4e088d" containerName="revoke-keys"
Oct 31 03:25:37.987422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:37.987354    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3935c86a-5f18-4ade-83a0-ccddfe4e088d" containerName="revoke-keys"
Oct 31 03:25:37.987456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:37.987444    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3935c86a-5f18-4ade-83a0-ccddfe4e088d" containerName="revoke-keys"
Oct 31 03:25:37.991221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:37.991202    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073317    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073362    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073470    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073551    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073676    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073732    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxb2\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073810    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.073867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.073852    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.098206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.098166    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:25:38.174710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174672    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.174901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174751    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.174901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174802    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.174901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174864    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174931    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.174971    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175003    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxb2\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175062    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175096    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175284    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175332    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175436    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175616    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.175890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.175874    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.177820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.177797    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.178061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.178042    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.231555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.231481    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.231716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.231574    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxb2\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.306708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.306618    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:38.734431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.734396    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:25:38.770405 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:38.770380    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0030d86d_0aa5_4123_b0f6_340c2877e30a.slice/crio-75e1608b8978ee166bd79201f01070082da529d8d36057287a4673062890f61c WatchSource:0}: Error finding container 75e1608b8978ee166bd79201f01070082da529d8d36057287a4673062890f61c: Status 404 returned error can't find the container with id 75e1608b8978ee166bd79201f01070082da529d8d36057287a4673062890f61c
Oct 31 03:25:38.772979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.772945    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:38.773085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.773013    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:38.773085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:38.773042    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:25:39.313339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:39.313301    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" event={"ID":"0030d86d-0aa5-4123-b0f6-340c2877e30a","Type":"ContainerStarted","Data":"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"}
Oct 31 03:25:39.313833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:39.313360    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" event={"ID":"0030d86d-0aa5-4123-b0f6-340c2877e30a","Type":"ContainerStarted","Data":"75e1608b8978ee166bd79201f01070082da529d8d36057287a4673062890f61c"}
Oct 31 03:25:39.427881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:39.427827    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podStartSLOduration=2.427812703 podStartE2EDuration="2.427812703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:39.425879227 +0000 UTC m=+1089.440440934" watchObservedRunningTime="2026-10-31 03:25:39.427812703 +0000 UTC m=+1089.442374410"
Oct 31 03:25:40.306832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:40.306794    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:40.308854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:40.308825    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:40.309032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:40.308873    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:41.308838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:41.308798    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:41.309494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:41.309449    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:42.307265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:42.307230    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:42.307459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:42.307295    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:43.001310 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:25:43.001266    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:25:43.307115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:43.307023    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:43.307404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:43.307372    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:44.013591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.013479    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"]
Oct 31 03:25:44.018699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.018677    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.052208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.052177    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"]
Oct 31 03:25:44.074506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.074460    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.074960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.074927    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnq6\" (UniqueName: \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.075075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.075058    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.112571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.096690    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:25:44.113668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.113635    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:25:44.114886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.113894    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:25:44.115053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.114684    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:25:44.176076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.176049    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.176232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.176201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.176307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.176262    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnq6\" (UniqueName: \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.176811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.176787    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.203005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.202971    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.231268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.231226    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnq6\" (UniqueName: \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\") pod \"maas-api-revoke-sub-41595c5b-cm6sd\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.312044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.311948    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:44.313035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.313002    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:44.337123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.336902    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:44.756050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:44.756022    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"]
Oct 31 03:25:44.757037 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:44.757010    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde98f2_edd5_4ead_8716_92a61b3b6953.slice/crio-285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3 WatchSource:0}: Error finding container 285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3: Status 404 returned error can't find the container with id 285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3
Oct 31 03:25:45.308113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.308001    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 03:25:45.308113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.308071    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 03:25:45.361937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.361906    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd" event={"ID":"cdde98f2-edd5-4ead-8716-92a61b3b6953","Type":"ContainerStarted","Data":"b84c814da335f3d9de7ec676335c4b5726b48f32b4249594597a74257c91e2dd"}
Oct 31 03:25:45.362118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.361963    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd" event={"ID":"cdde98f2-edd5-4ead-8716-92a61b3b6953","Type":"ContainerStarted","Data":"285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3"}
Oct 31 03:25:45.468133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.468070    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd" podStartSLOduration=2.468052332 podStartE2EDuration="2.468052332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:45.467289611 +0000 UTC m=+1095.481851320" watchObservedRunningTime="2026-10-31 03:25:45.468052332 +0000 UTC m=+1095.482614041"
Oct 31 03:25:45.817580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.817522    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"]
Oct 31 03:25:45.822815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.822790    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:45.889872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.889835    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"]
Oct 31 03:25:45.929540 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.929420    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:45.929940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.929915    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:45.930208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:45.930188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cf8\" (UniqueName: \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.031317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.031279    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.031492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.031427    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.031492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.031486    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cf8\" (UniqueName: \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.032291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.032216    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.034227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.034199    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.087070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.086987    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cf8\" (UniqueName: \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\") pod \"maas-api-revoke-sub-cff8be89-mxtrh\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.138878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.138844    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:46.313909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.313868    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:46.315674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.315651    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:46.316136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.316122    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:25:46.370793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.370717    5831 generic.go:356] "Generic (PLEG): container finished" podID="cdde98f2-edd5-4ead-8716-92a61b3b6953" containerID="b84c814da335f3d9de7ec676335c4b5726b48f32b4249594597a74257c91e2dd" exitCode=0
Oct 31 03:25:46.370914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.370788    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd" event={"ID":"cdde98f2-edd5-4ead-8716-92a61b3b6953","Type":"ContainerDied","Data":"b84c814da335f3d9de7ec676335c4b5726b48f32b4249594597a74257c91e2dd"}
Oct 31 03:25:46.536420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:46.536390    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"]
Oct 31 03:25:46.537798 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:25:46.537771    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bc629d_37c0_4aa6_9bf5_b6554f898b50.slice/crio-ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260 WatchSource:0}: Error finding container ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260: Status 404 returned error can't find the container with id ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260
Oct 31 03:25:47.381030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.380982    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh" event={"ID":"82bc629d-37c0-4aa6-9bf5-b6554f898b50","Type":"ContainerStarted","Data":"6be5dbbf299f7ef3093e7e2b6ec3761b091aa6337aa91253420e7176b6f7a9dc"}
Oct 31 03:25:47.381560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.381064    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh" event={"ID":"82bc629d-37c0-4aa6-9bf5-b6554f898b50","Type":"ContainerStarted","Data":"ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260"}
Oct 31 03:25:47.500442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.500271    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh" podStartSLOduration=2.5002501390000003 podStartE2EDuration="2.500250139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:47.488914166 +0000 UTC m=+1097.503475876" watchObservedRunningTime="2026-10-31 03:25:47.500250139 +0000 UTC m=+1097.514811847"
Oct 31 03:25:47.578992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.578964    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:47.679670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.679643    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\") pod \"cdde98f2-edd5-4ead-8716-92a61b3b6953\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") "
Oct 31 03:25:47.679859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.679678    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\") pod \"cdde98f2-edd5-4ead-8716-92a61b3b6953\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") "
Oct 31 03:25:47.679859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.679744    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\" (UniqueName: \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\") pod \"cdde98f2-edd5-4ead-8716-92a61b3b6953\" (UID: \"cdde98f2-edd5-4ead-8716-92a61b3b6953\") "
Oct 31 03:25:47.680064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.680038    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca" pod "cdde98f2-edd5-4ead-8716-92a61b3b6953" (UID: "cdde98f2-edd5-4ead-8716-92a61b3b6953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:25:47.680452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.680357    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:47.681996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.681971    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca" pod "cdde98f2-edd5-4ead-8716-92a61b3b6953" (UID: "cdde98f2-edd5-4ead-8716-92a61b3b6953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:47.682099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.682023    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6" pod "cdde98f2-edd5-4ead-8716-92a61b3b6953" (UID: "cdde98f2-edd5-4ead-8716-92a61b3b6953"). InnerVolumeSpecName "kube-api-access-7rnq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:47.781471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.781436    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde98f2-edd5-4ead-8716-92a61b3b6953-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:47.781471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:47.781465    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnq6\" (UniqueName: \"kubernetes.io/projected/cdde98f2-edd5-4ead-8716-92a61b3b6953-kube-api-access-7rnq6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:48.390132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:48.390095    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"
Oct 31 03:25:48.390600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:48.390095    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd" event={"ID":"cdde98f2-edd5-4ead-8716-92a61b3b6953","Type":"ContainerDied","Data":"285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3"}
Oct 31 03:25:48.390600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:48.390233    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285c663503be00aaef1d23be2b49f081221fe7e376252b6e5b9621b95091e3b3"
Oct 31 03:25:48.391816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:48.391789    5831 generic.go:356] "Generic (PLEG): container finished" podID="82bc629d-37c0-4aa6-9bf5-b6554f898b50" containerID="6be5dbbf299f7ef3093e7e2b6ec3761b091aa6337aa91253420e7176b6f7a9dc" exitCode=0
Oct 31 03:25:48.391953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:48.391886    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh" event={"ID":"82bc629d-37c0-4aa6-9bf5-b6554f898b50","Type":"ContainerDied","Data":"6be5dbbf299f7ef3093e7e2b6ec3761b091aa6337aa91253420e7176b6f7a9dc"}
Oct 31 03:25:50.571490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.571404    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh" event={"ID":"82bc629d-37c0-4aa6-9bf5-b6554f898b50","Type":"ContainerDied","Data":"ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260"}
Oct 31 03:25:50.572097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.572076    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3f0aac1e027b087fe4d94fcd277d3a115a36c632e07f11fe9dd14cf17b7260"
Oct 31 03:25:50.575426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.575400    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:50.868610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.857374    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\") pod \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") "
Oct 31 03:25:50.876926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.876885    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\" (UniqueName: \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\") pod \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") "
Oct 31 03:25:50.877103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.877033    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\") pod \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\" (UID: \"82bc629d-37c0-4aa6-9bf5-b6554f898b50\") "
Oct 31 03:25:50.878104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.861878    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca" pod "82bc629d-37c0-4aa6-9bf5-b6554f898b50" (UID: "82bc629d-37c0-4aa6-9bf5-b6554f898b50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:25:50.896750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.896649    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca" pod "82bc629d-37c0-4aa6-9bf5-b6554f898b50" (UID: "82bc629d-37c0-4aa6-9bf5-b6554f898b50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:50.901743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.901709    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8" pod "82bc629d-37c0-4aa6-9bf5-b6554f898b50" (UID: "82bc629d-37c0-4aa6-9bf5-b6554f898b50"). InnerVolumeSpecName "kube-api-access-k9cf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:50.978581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.978514    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:50.978581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.978562    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bc629d-37c0-4aa6-9bf5-b6554f898b50-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:50.978581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:50.978574    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cf8\" (UniqueName: \"kubernetes.io/projected/82bc629d-37c0-4aa6-9bf5-b6554f898b50-kube-api-access-k9cf8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.578960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:51.578926    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"
Oct 31 03:25:59.727950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.727909    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"]
Oct 31 03:25:59.728750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728726    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cdde98f2-edd5-4ead-8716-92a61b3b6953" containerName="revoke-keys"
Oct 31 03:25:59.728750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728750    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdde98f2-edd5-4ead-8716-92a61b3b6953" containerName="revoke-keys"
Oct 31 03:25:59.728955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728782    5831 cpu_manager.go:414] "containerMap: removing container" podUID="82bc629d-37c0-4aa6-9bf5-b6554f898b50" containerName="revoke-keys"
Oct 31 03:25:59.728955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728790    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bc629d-37c0-4aa6-9bf5-b6554f898b50" containerName="revoke-keys"
Oct 31 03:25:59.729031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728966    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdde98f2-edd5-4ead-8716-92a61b3b6953" containerName="revoke-keys"
Oct 31 03:25:59.729031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.728980    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bc629d-37c0-4aa6-9bf5-b6554f898b50" containerName="revoke-keys"
Oct 31 03:25:59.734096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.734080    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.786393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.786361    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"]
Oct 31 03:25:59.794877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.794839    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.795045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.794894    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhnm\" (UniqueName: \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.795180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.795159    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.826996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.826947    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:25:59.826996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.826954    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:25:59.827239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.827043    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:25:59.828668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.828647    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:25:59.896760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.896729    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.896971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.896783    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.896971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.896802    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhnm\" (UniqueName: \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.897542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.897507    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.899298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.899278    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:25:59.950360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:25:59.950323    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhnm\" (UniqueName: \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\") pod \"maas-api-revoke-sub-516fa178-548kn\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:26:00.048755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:00.048656    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:26:00.436504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:00.436470    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"]
Oct 31 03:26:00.438026 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:00.438000    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14709779_5f9a_46f0_8083_453b0a892736.slice/crio-3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b WatchSource:0}: Error finding container 3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b: Status 404 returned error can't find the container with id 3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b
Oct 31 03:26:00.655020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:00.654984    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn" event={"ID":"14709779-5f9a-46f0-8083-453b0a892736","Type":"ContainerStarted","Data":"3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b"}
Oct 31 03:26:01.663795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:01.663713    5831 generic.go:356] "Generic (PLEG): container finished" podID="14709779-5f9a-46f0-8083-453b0a892736" containerID="a8514f9062a3ada6e6c47e9b432da7d9e91f737907863c8acf1dcd186c2a14e4" exitCode=0
Oct 31 03:26:01.664160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:01.663802    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn" event={"ID":"14709779-5f9a-46f0-8083-453b0a892736","Type":"ContainerDied","Data":"a8514f9062a3ada6e6c47e9b432da7d9e91f737907863c8acf1dcd186c2a14e4"}
Oct 31 03:26:01.950172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:01.950135    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"]
Oct 31 03:26:01.954501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:01.954479    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.008234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.008204    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"]
Oct 31 03:26:02.026605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.026575    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.026796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.026773    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5nk\" (UniqueName: \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.026879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.026826    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.128062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.128022    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.128252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.128140    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5nk\" (UniqueName: \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.128252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.128174    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.128734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.128705    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.130786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.130760    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.182267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.182234    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5nk\" (UniqueName: \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\") pod \"maas-api-revoke-sub-580517da-lckr6\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.268257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.268172    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:02.650739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.650706    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"]
Oct 31 03:26:02.651680 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:02.651654    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c8bfe6_beab_4493_9f4f_3e98944d2b5b.slice/crio-f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5 WatchSource:0}: Error finding container f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5: Status 404 returned error can't find the container with id f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5
Oct 31 03:26:02.673685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.673658    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6" event={"ID":"41c8bfe6-beab-4493-9f4f-3e98944d2b5b","Type":"ContainerStarted","Data":"f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5"}
Oct 31 03:26:02.809793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.809766    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:26:02.939967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.939929    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\" (UniqueName: \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\") pod \"14709779-5f9a-46f0-8083-453b0a892736\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") "
Oct 31 03:26:02.940142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.940066    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\") pod \"14709779-5f9a-46f0-8083-453b0a892736\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") "
Oct 31 03:26:02.940142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.940097    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\") pod \"14709779-5f9a-46f0-8083-453b0a892736\" (UID: \"14709779-5f9a-46f0-8083-453b0a892736\") "
Oct 31 03:26:02.940675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.940633    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca" pod "14709779-5f9a-46f0-8083-453b0a892736" (UID: "14709779-5f9a-46f0-8083-453b0a892736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:02.942573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.942520    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm" pod "14709779-5f9a-46f0-8083-453b0a892736" (UID: "14709779-5f9a-46f0-8083-453b0a892736"). InnerVolumeSpecName "kube-api-access-jfhnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:02.943087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:02.943062    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca" pod "14709779-5f9a-46f0-8083-453b0a892736" (UID: "14709779-5f9a-46f0-8083-453b0a892736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:03.042407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.042371    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhnm\" (UniqueName: \"kubernetes.io/projected/14709779-5f9a-46f0-8083-453b0a892736-kube-api-access-jfhnm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:03.042407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.042405    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14709779-5f9a-46f0-8083-453b0a892736-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:03.042654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.042420    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14709779-5f9a-46f0-8083-453b0a892736-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:03.683336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.683303    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"
Oct 31 03:26:03.683862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.683302    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn" event={"ID":"14709779-5f9a-46f0-8083-453b0a892736","Type":"ContainerDied","Data":"3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b"}
Oct 31 03:26:03.683862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.683459    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3603ba9552efb85fcd4c4495079bb4c053e3e65f1bb246660849b77c48ae8d2b"
Oct 31 03:26:03.685367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.685339    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6" event={"ID":"41c8bfe6-beab-4493-9f4f-3e98944d2b5b","Type":"ContainerStarted","Data":"a75fbf052b7f9b1c21b2dd79729b70e664715aaa6862c80895edb4c9983f9548"}
Oct 31 03:26:03.798080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:03.798029    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6" podStartSLOduration=2.798012479 podStartE2EDuration="2.798012479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:03.794433098 +0000 UTC m=+1113.808994807" watchObservedRunningTime="2026-10-31 03:26:03.798012479 +0000 UTC m=+1113.812574186"
Oct 31 03:26:04.696820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:04.696781    5831 generic.go:356] "Generic (PLEG): container finished" podID="41c8bfe6-beab-4493-9f4f-3e98944d2b5b" containerID="a75fbf052b7f9b1c21b2dd79729b70e664715aaa6862c80895edb4c9983f9548" exitCode=0
Oct 31 03:26:04.697292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:04.696862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6" event={"ID":"41c8bfe6-beab-4493-9f4f-3e98944d2b5b","Type":"ContainerDied","Data":"a75fbf052b7f9b1c21b2dd79729b70e664715aaa6862c80895edb4c9983f9548"}
Oct 31 03:26:05.960074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:05.960016    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:06.038591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.038557    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\") pod \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") "
Oct 31 03:26:06.038766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.038666    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\") pod \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") "
Oct 31 03:26:06.038766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.038706    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\" (UniqueName: \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\") pod \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\" (UID: \"41c8bfe6-beab-4493-9f4f-3e98944d2b5b\") "
Oct 31 03:26:06.039018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.038994    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca" pod "41c8bfe6-beab-4493-9f4f-3e98944d2b5b" (UID: "41c8bfe6-beab-4493-9f4f-3e98944d2b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:06.041055 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.041025    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca" pod "41c8bfe6-beab-4493-9f4f-3e98944d2b5b" (UID: "41c8bfe6-beab-4493-9f4f-3e98944d2b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:06.041162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.041049    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk" pod "41c8bfe6-beab-4493-9f4f-3e98944d2b5b" (UID: "41c8bfe6-beab-4493-9f4f-3e98944d2b5b"). InnerVolumeSpecName "kube-api-access-6h5nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:06.140452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.140417    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:06.140452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.140452    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:06.140681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.140468    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5nk\" (UniqueName: \"kubernetes.io/projected/41c8bfe6-beab-4493-9f4f-3e98944d2b5b-kube-api-access-6h5nk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:06.468397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.468365    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:26:06.712966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.712935    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"
Oct 31 03:26:06.713136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.712971    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6" event={"ID":"41c8bfe6-beab-4493-9f4f-3e98944d2b5b","Type":"ContainerDied","Data":"f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5"}
Oct 31 03:26:06.713136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:06.713018    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b2939fdbce3ad8afe8a5d258a7504ae7d99c3965613bdecaf6f3ed19fc99d5"
Oct 31 03:26:12.978723 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:26:12.978681    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:26:15.401835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.401795    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"]
Oct 31 03:26:15.402476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402458    5831 cpu_manager.go:414] "containerMap: removing container" podUID="14709779-5f9a-46f0-8083-453b0a892736" containerName="revoke-keys"
Oct 31 03:26:15.402563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402479    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14709779-5f9a-46f0-8083-453b0a892736" containerName="revoke-keys"
Oct 31 03:26:15.402563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402494    5831 cpu_manager.go:414] "containerMap: removing container" podUID="41c8bfe6-beab-4493-9f4f-3e98944d2b5b" containerName="revoke-keys"
Oct 31 03:26:15.402563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402498    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c8bfe6-beab-4493-9f4f-3e98944d2b5b" containerName="revoke-keys"
Oct 31 03:26:15.402705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402658    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="14709779-5f9a-46f0-8083-453b0a892736" containerName="revoke-keys"
Oct 31 03:26:15.402705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.402668    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c8bfe6-beab-4493-9f4f-3e98944d2b5b" containerName="revoke-keys"
Oct 31 03:26:15.407347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.407330    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.460030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.460002    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"]
Oct 31 03:26:15.500866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.500786    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:26:15.500866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.500809    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:26:15.501113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.500896    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:26:15.501777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.501752    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:26:15.526606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.526581    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.526745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.526625    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.526797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.526782    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbg6\" (UniqueName: \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.627821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.627791    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.628002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.627922    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbg6\" (UniqueName: \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.628054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.628036    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.628708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.628675    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.630321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.630298    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.682919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.682890    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbg6\" (UniqueName: \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\") pod \"maas-api-revoke-sub-71525716-jjkk5\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:15.720875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:15.720841    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:16.152277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:16.152247    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"]
Oct 31 03:26:16.154777 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:16.154746    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f414b21_bbc5_4b39_af24_6520f6a1f325.slice/crio-d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7 WatchSource:0}: Error finding container d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7: Status 404 returned error can't find the container with id d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7
Oct 31 03:26:16.824189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:16.824086    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5" event={"ID":"5f414b21-bbc5-4b39-af24-6520f6a1f325","Type":"ContainerStarted","Data":"7a527a22e5d8d1e07eb9e4096149e3be1eb8553c9211ed04f4014edee513721a"}
Oct 31 03:26:16.825007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:16.824193    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5" event={"ID":"5f414b21-bbc5-4b39-af24-6520f6a1f325","Type":"ContainerStarted","Data":"d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7"}
Oct 31 03:26:16.937173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:16.937113    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5" podStartSLOduration=1.93709434 podStartE2EDuration="1.93709434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:16.9361006 +0000 UTC m=+1126.950662309" watchObservedRunningTime="2026-10-31 03:26:16.93709434 +0000 UTC m=+1126.951656049"
Oct 31 03:26:17.833041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:17.833006    5831 generic.go:356] "Generic (PLEG): container finished" podID="5f414b21-bbc5-4b39-af24-6520f6a1f325" containerID="7a527a22e5d8d1e07eb9e4096149e3be1eb8553c9211ed04f4014edee513721a" exitCode=0
Oct 31 03:26:17.833597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:17.833084    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5" event={"ID":"5f414b21-bbc5-4b39-af24-6520f6a1f325","Type":"ContainerDied","Data":"7a527a22e5d8d1e07eb9e4096149e3be1eb8553c9211ed04f4014edee513721a"}
Oct 31 03:26:18.507113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.507076    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"]
Oct 31 03:26:18.511114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.511092    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.565567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.565518    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"]
Oct 31 03:26:18.574281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.574248    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5jw\" (UniqueName: \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.574542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.574503    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.574644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.574620    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.675904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.675873    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.676085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.675914    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.676085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.676008    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5jw\" (UniqueName: \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.676732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.676653    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.678419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.678397    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.730005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.729964    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5jw\" (UniqueName: \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\") pod \"maas-api-revoke-sub-10762baa-jnzjh\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:18.825655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:18.825573    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:19.058676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.058651    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:19.183902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.183878    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\") pod \"5f414b21-bbc5-4b39-af24-6520f6a1f325\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") "
Oct 31 03:26:19.184182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.184158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\") pod \"5f414b21-bbc5-4b39-af24-6520f6a1f325\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") "
Oct 31 03:26:19.184248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.184227    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\" (UniqueName: \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\") pod \"5f414b21-bbc5-4b39-af24-6520f6a1f325\" (UID: \"5f414b21-bbc5-4b39-af24-6520f6a1f325\") "
Oct 31 03:26:19.184309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.184270    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca" pod "5f414b21-bbc5-4b39-af24-6520f6a1f325" (UID: "5f414b21-bbc5-4b39-af24-6520f6a1f325"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:19.184649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.184628    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.186366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.186340    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca" pod "5f414b21-bbc5-4b39-af24-6520f6a1f325" (UID: "5f414b21-bbc5-4b39-af24-6520f6a1f325"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:19.186470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.186438    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6" pod "5f414b21-bbc5-4b39-af24-6520f6a1f325" (UID: "5f414b21-bbc5-4b39-af24-6520f6a1f325"). InnerVolumeSpecName "kube-api-access-dpbg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:19.215746 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:19.215700    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7a1fe8_f84f_44a1_aaca_a2df9dd87d38.slice/crio-7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201 WatchSource:0}: Error finding container 7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201: Status 404 returned error can't find the container with id 7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201
Oct 31 03:26:19.219698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.219672    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"]
Oct 31 03:26:19.286418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.286391    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f414b21-bbc5-4b39-af24-6520f6a1f325-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.286418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.286420    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbg6\" (UniqueName: \"kubernetes.io/projected/5f414b21-bbc5-4b39-af24-6520f6a1f325-kube-api-access-dpbg6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:19.848917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.848827    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"
Oct 31 03:26:19.848917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.848858    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5" event={"ID":"5f414b21-bbc5-4b39-af24-6520f6a1f325","Type":"ContainerDied","Data":"d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7"}
Oct 31 03:26:19.848917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.848911    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d327057f2d70cc66d431a62cf96800d9fbda173971234607c1d1cea140bf5ea7"
Oct 31 03:26:19.850773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.850744    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh" event={"ID":"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38","Type":"ContainerStarted","Data":"9928b08645e111637afe900b6cf13bc40cb96eab844f3e667163cdf8fccd555f"}
Oct 31 03:26:19.850934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.850795    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh" event={"ID":"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38","Type":"ContainerStarted","Data":"7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201"}
Oct 31 03:26:19.961690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:19.961624    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh" podStartSLOduration=1.9616106370000002 podStartE2EDuration="1.961610637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:19.957486946 +0000 UTC m=+1129.972048653" watchObservedRunningTime="2026-10-31 03:26:19.961610637 +0000 UTC m=+1129.976172345"
Oct 31 03:26:20.869816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:20.869759    5831 generic.go:356] "Generic (PLEG): container finished" podID="3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" containerID="9928b08645e111637afe900b6cf13bc40cb96eab844f3e667163cdf8fccd555f" exitCode=0
Oct 31 03:26:20.870338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:20.869991    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh" event={"ID":"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38","Type":"ContainerDied","Data":"9928b08645e111637afe900b6cf13bc40cb96eab844f3e667163cdf8fccd555f"}
Oct 31 03:26:22.041460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.041431    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:22.125515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.125477    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\" (UniqueName: \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\") pod \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") "
Oct 31 03:26:22.125690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.125562    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\") pod \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") "
Oct 31 03:26:22.125771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.125755    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\") pod \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\" (UID: \"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38\") "
Oct 31 03:26:22.126160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.126125    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca" pod "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" (UID: "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:22.126463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.126256    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:22.128028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.127962    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw" pod "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" (UID: "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38"). InnerVolumeSpecName "kube-api-access-9x5jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:22.128135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.128052    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca" pod "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" (UID: "3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:22.227363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.227280    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5jw\" (UniqueName: \"kubernetes.io/projected/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-kube-api-access-9x5jw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:22.227363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.227317    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:22.887097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.887063    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh" event={"ID":"3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38","Type":"ContainerDied","Data":"7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201"}
Oct 31 03:26:22.887097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.887091    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"
Oct 31 03:26:22.887317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:22.887121    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7beb473baf3718728974382cd04dbd1fa873194928cbb21ee56494e7347a5201"
Oct 31 03:26:28.918556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:28.915627    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:26:28.918556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:28.915699    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:29.454040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.454003    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"]
Oct 31 03:26:29.454899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.454874    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" containerName="revoke-keys"
Oct 31 03:26:29.454899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.454897    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" containerName="revoke-keys"
Oct 31 03:26:29.455067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.454922    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5f414b21-bbc5-4b39-af24-6520f6a1f325" containerName="revoke-keys"
Oct 31 03:26:29.455067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.454927    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f414b21-bbc5-4b39-af24-6520f6a1f325" containerName="revoke-keys"
Oct 31 03:26:29.455067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.455040    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" containerName="revoke-keys"
Oct 31 03:26:29.455067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.455053    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f414b21-bbc5-4b39-af24-6520f6a1f325" containerName="revoke-keys"
Oct 31 03:26:29.459192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.459171    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.486450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.486424    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlv2\" (UniqueName: \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.486621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.486464    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.486621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.486559    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.513743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.513711    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"]
Oct 31 03:26:29.552572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.552520    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:26:29.552572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.552520    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:26:29.552791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.552583    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:26:29.555701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.555679    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:26:29.589177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.589136    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlv2\" (UniqueName: \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.589366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.589210    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.589366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.589269    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.590358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.590317    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.598585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.593829    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.652043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.652011    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlv2\" (UniqueName: \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\") pod \"maas-api-revoke-sub-cfbb973a-p2l44\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:29.774786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:29.774691    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:30.153272 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:30.153237    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d28a9b9_3f32_4f7c_b603_0d2ec7bd1d23.slice/crio-b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce WatchSource:0}: Error finding container b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce: Status 404 returned error can't find the container with id b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce
Oct 31 03:26:30.154041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:30.154017    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"]
Oct 31 03:26:31.074982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.074943    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44" event={"ID":"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23","Type":"ContainerStarted","Data":"8c85b5e0b9e26d6f800d99737da5d737d80af3af1e471ef2d618179e2debb126"}
Oct 31 03:26:31.075186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.075000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44" event={"ID":"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23","Type":"ContainerStarted","Data":"b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce"}
Oct 31 03:26:31.183155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.183104    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44" podStartSLOduration=2.183087078 podStartE2EDuration="2.183087078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:31.181216026 +0000 UTC m=+1141.195777734" watchObservedRunningTime="2026-10-31 03:26:31.183087078 +0000 UTC m=+1141.197648787"
Oct 31 03:26:31.306105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.306068    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:26:31.310464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.310443    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.396218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.396133    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:26:31.409818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.409785    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.409869    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.409901    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410013    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410075    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxxp\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410110    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410149    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410169    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.410397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.410304    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512011    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512117    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512137    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512151    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512260    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxxp\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512288    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512333    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512360    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512454    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512510    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512729    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.512889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.512856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.513085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.513057    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.514777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.514755    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.515098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.515078    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.565341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.565312    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.565486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.565362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxxp\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:31.624742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:31.624707    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:32.085106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.085012    5831 generic.go:356] "Generic (PLEG): container finished" podID="5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" containerID="8c85b5e0b9e26d6f800d99737da5d737d80af3af1e471ef2d618179e2debb126" exitCode=0
Oct 31 03:26:32.085106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.085074    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44" event={"ID":"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23","Type":"ContainerDied","Data":"8c85b5e0b9e26d6f800d99737da5d737d80af3af1e471ef2d618179e2debb126"}
Oct 31 03:26:32.119635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.119606    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:26:32.121162 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:32.121135    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394efdbe_e6c8_4c26_9f74_01ae047efee5.slice/crio-2ca5cc3cf2d75a85c8478081811477de9625600901a52a290de6fb63a5252a7c WatchSource:0}: Error finding container 2ca5cc3cf2d75a85c8478081811477de9625600901a52a290de6fb63a5252a7c: Status 404 returned error can't find the container with id 2ca5cc3cf2d75a85c8478081811477de9625600901a52a290de6fb63a5252a7c
Oct 31 03:26:32.123842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.123805    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:32.123910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.123892    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:32.123973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:32.123958    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:33.111183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.111142    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" event={"ID":"394efdbe-e6c8-4c26-9f74-01ae047efee5","Type":"ContainerStarted","Data":"dcf60afd41e9ba7bd2adad581603a3cea2d17cf6330b34dd53f18a71c18724c9"}
Oct 31 03:26:33.112319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.112287    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" event={"ID":"394efdbe-e6c8-4c26-9f74-01ae047efee5","Type":"ContainerStarted","Data":"2ca5cc3cf2d75a85c8478081811477de9625600901a52a290de6fb63a5252a7c"}
Oct 31 03:26:33.344298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.344272    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:33.351675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.351629    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podStartSLOduration=2.351615194 podStartE2EDuration="2.351615194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:33.303060092 +0000 UTC m=+1143.317621831" watchObservedRunningTime="2026-10-31 03:26:33.351615194 +0000 UTC m=+1143.366176904"
Oct 31 03:26:33.442890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.442861    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\") pod \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") "
Oct 31 03:26:33.443086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.442937    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\" (UniqueName: \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\") pod \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") "
Oct 31 03:26:33.443086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.443041    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\") pod \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\" (UID: \"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23\") "
Oct 31 03:26:33.443397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.443371    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca" pod "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" (UID: "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:33.443677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.443660    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:33.445176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.445151    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca" pod "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" (UID: "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:33.445243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.445153    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2" pod "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" (UID: "5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23"). InnerVolumeSpecName "kube-api-access-9zlv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:33.544663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.544630    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:33.544663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.544662    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zlv2\" (UniqueName: \"kubernetes.io/projected/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23-kube-api-access-9zlv2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:33.625880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.625833    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:33.627519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.627490    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:33.627680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:33.627559    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:34.122984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.122953    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"
Oct 31 03:26:34.122984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.122970    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44" event={"ID":"5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23","Type":"ContainerDied","Data":"b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce"}
Oct 31 03:26:34.123485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.123033    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b738c24f81195dd8aff160eb0fb73113b8c4fef58ba341e2164337d838f952ce"
Oct 31 03:26:34.214586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.214552    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"]
Oct 31 03:26:34.215191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.215177    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" containerName="revoke-keys"
Oct 31 03:26:34.215238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.215193    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" containerName="revoke-keys"
Oct 31 03:26:34.215299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.215290    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" containerName="revoke-keys"
Oct 31 03:26:34.219111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.219090    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.254517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.254485    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"]
Oct 31 03:26:34.255075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.255050    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.255210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.255193    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.255281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.255241    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp8n\" (UniqueName: \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.268472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.268406    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:26:34.268651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.268521    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:26:34.268651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.268586    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:26:34.268651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.268641    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:26:34.357464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.357414    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.357695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.357498    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp8n\" (UniqueName: \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.357780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.357694    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.358560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.358498    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.362511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.362450    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.415571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.415461    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp8n\" (UniqueName: \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\") pod \"maas-api-revoke-sub-846fcd47-m8pnq\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.544414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.544381    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:34.630044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.629978    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:34.630206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.630037    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:34.942706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:34.942671    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"]
Oct 31 03:26:34.943248 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:34.943205    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7800029_029a_415f_8938_62dbde9fb104.slice/crio-180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab WatchSource:0}: Error finding container 180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab: Status 404 returned error can't find the container with id 180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab
Oct 31 03:26:35.133786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:35.133752    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq" event={"ID":"c7800029-029a-415f-8938-62dbde9fb104","Type":"ContainerStarted","Data":"180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab"}
Oct 31 03:26:35.625732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:35.625634    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:35.625732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:35.625696    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:36.143546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:36.143496    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq" event={"ID":"c7800029-029a-415f-8938-62dbde9fb104","Type":"ContainerStarted","Data":"4ba5c36af420d3a16bec5582a6ea6f10a2d3125f061f11ab888745d5e0524adf"}
Oct 31 03:26:36.269137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:36.269080    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq" podStartSLOduration=2.269061321 podStartE2EDuration="2.269061321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:36.265404805 +0000 UTC m=+1146.279966514" watchObservedRunningTime="2026-10-31 03:26:36.269061321 +0000 UTC m=+1146.283623030"
Oct 31 03:26:36.625164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:36.625126    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:36.625383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:36.625185    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:37.153235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:37.153205    5831 generic.go:356] "Generic (PLEG): container finished" podID="c7800029-029a-415f-8938-62dbde9fb104" containerID="4ba5c36af420d3a16bec5582a6ea6f10a2d3125f061f11ab888745d5e0524adf" exitCode=0
Oct 31 03:26:37.153667 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:37.153256    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq" event={"ID":"c7800029-029a-415f-8938-62dbde9fb104","Type":"ContainerDied","Data":"4ba5c36af420d3a16bec5582a6ea6f10a2d3125f061f11ab888745d5e0524adf"}
Oct 31 03:26:37.625944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:37.625904    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:37.626140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:37.625968    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:38.368671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.368641    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:38.437750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.437713    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\") pod \"c7800029-029a-415f-8938-62dbde9fb104\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") "
Oct 31 03:26:38.438024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.437835    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\") pod \"c7800029-029a-415f-8938-62dbde9fb104\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") "
Oct 31 03:26:38.438024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.437869    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\" (UniqueName: \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\") pod \"c7800029-029a-415f-8938-62dbde9fb104\" (UID: \"c7800029-029a-415f-8938-62dbde9fb104\") "
Oct 31 03:26:38.438179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.438138    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca" pod "c7800029-029a-415f-8938-62dbde9fb104" (UID: "c7800029-029a-415f-8938-62dbde9fb104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:38.438310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.438293    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7800029-029a-415f-8938-62dbde9fb104-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:38.440161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.440131    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca" pod "c7800029-029a-415f-8938-62dbde9fb104" (UID: "c7800029-029a-415f-8938-62dbde9fb104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:38.440297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.440139    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n" pod "c7800029-029a-415f-8938-62dbde9fb104" (UID: "c7800029-029a-415f-8938-62dbde9fb104"). InnerVolumeSpecName "kube-api-access-bjp8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:38.533616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.533573    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:26:38.534209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.534191    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c7800029-029a-415f-8938-62dbde9fb104" containerName="revoke-keys"
Oct 31 03:26:38.534209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.534207    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7800029-029a-415f-8938-62dbde9fb104" containerName="revoke-keys"
Oct 31 03:26:38.534367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.534294    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7800029-029a-415f-8938-62dbde9fb104" containerName="revoke-keys"
Oct 31 03:26:38.537924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.537904    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.539346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.539324    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7800029-029a-415f-8938-62dbde9fb104-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:38.539565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.539503    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjp8n\" (UniqueName: \"kubernetes.io/projected/c7800029-029a-415f-8938-62dbde9fb104-kube-api-access-bjp8n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:38.600199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.600151    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:26:38.626961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.626925    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:26:38.627720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.626985    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:26:38.640596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640565    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.640774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640616    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.640774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640640    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtdm\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.640774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640745    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.640936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640792    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.640995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.640944    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.641097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.641081    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.641154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.641119    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.641217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.641198    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742491    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742585    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742634    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742683    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742731    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742756    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtdm\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.742862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742789    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742864    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742962    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.742966    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.743094    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.743170    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.743568    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.743855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.743830    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.745176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.745149    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.745447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.745426    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.796945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.796905    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtdm\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.796945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.796932    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:38.855657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:38.855620    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:39.173389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.173310    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"
Oct 31 03:26:39.173389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.173326    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq" event={"ID":"c7800029-029a-415f-8938-62dbde9fb104","Type":"ContainerDied","Data":"180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab"}
Oct 31 03:26:39.173389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.173381    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180ad8edf8d0c8222ee3d17dd555f24ebd7e9d73255b1af95b5606c5f75a6eab"
Oct 31 03:26:39.287596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.287564    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:26:39.298678 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:39.298641    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18340bd6_ca06_4101_a97f_0f97f61e864a.slice/crio-e00f99efda1312101ff4c45493ebf94f0c3bcf6b4ee8d91678441b7192f13046 WatchSource:0}: Error finding container e00f99efda1312101ff4c45493ebf94f0c3bcf6b4ee8d91678441b7192f13046: Status 404 returned error can't find the container with id e00f99efda1312101ff4c45493ebf94f0c3bcf6b4ee8d91678441b7192f13046
Oct 31 03:26:39.302267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.301813    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:39.302267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.301909    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:39.302267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.301954    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:26:39.324650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:39.324054    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:26:40.202062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.202025    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:40.236408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.236353    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:40.236895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.236876    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:26:40.239599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.239497    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" event={"ID":"18340bd6-ca06-4101-a97f-0f97f61e864a","Type":"ContainerStarted","Data":"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"}
Oct 31 03:26:40.239599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.239561    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" event={"ID":"18340bd6-ca06-4101-a97f-0f97f61e864a","Type":"ContainerStarted","Data":"e00f99efda1312101ff4c45493ebf94f0c3bcf6b4ee8d91678441b7192f13046"}
Oct 31 03:26:40.655632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:40.652842    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podStartSLOduration=2.652819847 podStartE2EDuration="2.652819847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:40.534254964 +0000 UTC m=+1150.548816674" watchObservedRunningTime="2026-10-31 03:26:40.652819847 +0000 UTC m=+1150.667381557"
Oct 31 03:26:41.069560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:41.065031    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:41.107555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:41.105677    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:41.107555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:41.105743    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:41.863701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:41.863666    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:41.864312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:41.864282    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:42.238667 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:42.237628    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:42.873019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:42.872973    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:42.874436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:42.874411    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:42.875097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:42.875069    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:43.163613 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:26:43.162449    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:26:43.858732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:43.858679    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:43.859054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:43.859018    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:44.856673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:44.856632    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:44.857101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:44.856708    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:45.041435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.040091    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"]
Oct 31 03:26:45.049113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.049078    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.109973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.109883    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"]
Oct 31 03:26:45.125556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.124734    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.125556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.124811    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zqv\" (UniqueName: \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.125556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.124890    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.143228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.141927    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:26:45.143228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.142634    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:26:45.143228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.142686    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:26:45.151695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.151671    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:26:45.226302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.226269    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.226478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.226321    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zqv\" (UniqueName: \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.226478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.226377    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.227180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.227139    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.229342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.229321    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.312244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.312209    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zqv\" (UniqueName: \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\") pod \"maas-api-revoke-sub-5b340484-fhmpn\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.370755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.370672    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:45.818295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.818246    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"]
Oct 31 03:26:45.821998 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:45.821936    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b89131_2728_4288_ac7b_0c887f6a8a7e.slice/crio-598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b WatchSource:0}: Error finding container 598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b: Status 404 returned error can't find the container with id 598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b
Oct 31 03:26:45.856515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.856484    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:45.856650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:45.856549    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:46.550406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:46.550306    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn" event={"ID":"35b89131-2728-4288-ac7b-0c887f6a8a7e","Type":"ContainerStarted","Data":"b795d6686ecc078f2fb9028d3d04f267952f8d10d2b4a257b0a9e4de7b3c44ca"}
Oct 31 03:26:46.550406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:46.550372    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn" event={"ID":"35b89131-2728-4288-ac7b-0c887f6a8a7e","Type":"ContainerStarted","Data":"598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b"}
Oct 31 03:26:46.669513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:46.669455    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn" podStartSLOduration=2.669440874 podStartE2EDuration="2.669440874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:46.666367505 +0000 UTC m=+1156.680929214" watchObservedRunningTime="2026-10-31 03:26:46.669440874 +0000 UTC m=+1156.684002582"
Oct 31 03:26:46.856905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:46.856820    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:46.856905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:46.856871    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:47.559266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.559217    5831 generic.go:356] "Generic (PLEG): container finished" podID="35b89131-2728-4288-ac7b-0c887f6a8a7e" containerID="b795d6686ecc078f2fb9028d3d04f267952f8d10d2b4a257b0a9e4de7b3c44ca" exitCode=0
Oct 31 03:26:47.559690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.559294    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn" event={"ID":"35b89131-2728-4288-ac7b-0c887f6a8a7e","Type":"ContainerDied","Data":"b795d6686ecc078f2fb9028d3d04f267952f8d10d2b4a257b0a9e4de7b3c44ca"}
Oct 31 03:26:47.617845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.617814    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"]
Oct 31 03:26:47.624046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.624019    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.676762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.676728    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"]
Oct 31 03:26:47.759212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.759177    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjzm\" (UniqueName: \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.759383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.759339    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.759383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.759375    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.856294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.856206    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:26:47.856294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.856268    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 31 03:26:47.860896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.860872    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjzm\" (UniqueName: \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.861005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.860988    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.861063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.861026    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.861891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.861863    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.863544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.863506    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.915272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.915235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjzm\" (UniqueName: \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\") pod \"maas-api-revoke-sub-0952bd3f-tcxn7\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:47.943168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:47.943138    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:48.346555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.339085    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"]
Oct 31 03:26:48.346555 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:48.339443    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c6b06a_753f_4922_bb7c_d66063c5a790.slice/crio-c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3 WatchSource:0}: Error finding container c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3: Status 404 returned error can't find the container with id c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3
Oct 31 03:26:48.571692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.571652    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7" event={"ID":"28c6b06a-753f-4922-bb7c-d66063c5a790","Type":"ContainerStarted","Data":"c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3"}
Oct 31 03:26:48.733663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.733629    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:48.876031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.874836    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\" (UniqueName: \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\") pod \"35b89131-2728-4288-ac7b-0c887f6a8a7e\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") "
Oct 31 03:26:48.876031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.875001    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\") pod \"35b89131-2728-4288-ac7b-0c887f6a8a7e\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") "
Oct 31 03:26:48.876031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.875048    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\") pod \"35b89131-2728-4288-ac7b-0c887f6a8a7e\" (UID: \"35b89131-2728-4288-ac7b-0c887f6a8a7e\") "
Oct 31 03:26:48.876031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.875735    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca" pod "35b89131-2728-4288-ac7b-0c887f6a8a7e" (UID: "35b89131-2728-4288-ac7b-0c887f6a8a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:48.889089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.889043    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv" pod "35b89131-2728-4288-ac7b-0c887f6a8a7e" (UID: "35b89131-2728-4288-ac7b-0c887f6a8a7e"). InnerVolumeSpecName "kube-api-access-s8zqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:48.889677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.889628    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca" pod "35b89131-2728-4288-ac7b-0c887f6a8a7e" (UID: "35b89131-2728-4288-ac7b-0c887f6a8a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:48.976519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.976488    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:48.976519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.976518    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zqv\" (UniqueName: \"kubernetes.io/projected/35b89131-2728-4288-ac7b-0c887f6a8a7e-kube-api-access-s8zqv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:48.976519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:48.976549    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b89131-2728-4288-ac7b-0c887f6a8a7e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:49.585755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.585667    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"
Oct 31 03:26:49.585755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.585711    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn" event={"ID":"35b89131-2728-4288-ac7b-0c887f6a8a7e","Type":"ContainerDied","Data":"598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b"}
Oct 31 03:26:49.586317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.585770    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598f3ae6722fbfdde765c99fd5667266e618adcbdcdb0b87df7dab1e7deeed1b"
Oct 31 03:26:49.588058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.588024    5831 generic.go:356] "Generic (PLEG): container finished" podID="28c6b06a-753f-4922-bb7c-d66063c5a790" containerID="e61928e793f11215259e6de5bcdc3c9eda389d95a186c0dad064c7881f702032" exitCode=0
Oct 31 03:26:49.588220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.588119    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7" event={"ID":"28c6b06a-753f-4922-bb7c-d66063c5a790","Type":"ContainerDied","Data":"e61928e793f11215259e6de5bcdc3c9eda389d95a186c0dad064c7881f702032"}
Oct 31 03:26:49.897798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.897703    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:26:49.897798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:49.897774    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:50.010257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.010153    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:50.011173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.011149    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:50.012038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.012016    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:26:50.745042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.745018    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:50.901996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.901913    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\") pod \"28c6b06a-753f-4922-bb7c-d66063c5a790\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") "
Oct 31 03:26:50.901996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.901969    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\") pod \"28c6b06a-753f-4922-bb7c-d66063c5a790\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") "
Oct 31 03:26:50.902238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.902080    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\" (UniqueName: \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\") pod \"28c6b06a-753f-4922-bb7c-d66063c5a790\" (UID: \"28c6b06a-753f-4922-bb7c-d66063c5a790\") "
Oct 31 03:26:50.902394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.902370    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca" pod "28c6b06a-753f-4922-bb7c-d66063c5a790" (UID: "28c6b06a-753f-4922-bb7c-d66063c5a790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:50.902644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.902629    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:50.904143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.904112    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca" pod "28c6b06a-753f-4922-bb7c-d66063c5a790" (UID: "28c6b06a-753f-4922-bb7c-d66063c5a790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:50.904281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:50.904261    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm" pod "28c6b06a-753f-4922-bb7c-d66063c5a790" (UID: "28c6b06a-753f-4922-bb7c-d66063c5a790"). InnerVolumeSpecName "kube-api-access-7mjzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:51.004364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:51.004326    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c6b06a-753f-4922-bb7c-d66063c5a790-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:51.004364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:51.004356    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjzm\" (UniqueName: \"kubernetes.io/projected/28c6b06a-753f-4922-bb7c-d66063c5a790-kube-api-access-7mjzm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:26:51.605338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:51.605306    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"
Oct 31 03:26:51.605518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:51.605343    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7" event={"ID":"28c6b06a-753f-4922-bb7c-d66063c5a790","Type":"ContainerDied","Data":"c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3"}
Oct 31 03:26:51.605518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:51.605392    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c571186ed9a56466f87b68d3fd4e1ea275960a2881e03561a8ab8d71f80459b3"
Oct 31 03:26:56.055464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:56.055424    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"]
Oct 31 03:26:56.060074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:56.060046    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0785a91e-tcg68"]
Oct 31 03:26:56.136876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:56.136845    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4710e738-1fe7-43fc-977f-73920838acb9" path="/var/lib/kubelet/pods/4710e738-1fe7-43fc-977f-73920838acb9/volumes"
Oct 31 03:26:58.411063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.410986    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"]
Oct 31 03:26:58.411693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411676    5831 cpu_manager.go:414] "containerMap: removing container" podUID="35b89131-2728-4288-ac7b-0c887f6a8a7e" containerName="revoke-keys"
Oct 31 03:26:58.411749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411695    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b89131-2728-4288-ac7b-0c887f6a8a7e" containerName="revoke-keys"
Oct 31 03:26:58.411749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411709    5831 cpu_manager.go:414] "containerMap: removing container" podUID="28c6b06a-753f-4922-bb7c-d66063c5a790" containerName="revoke-keys"
Oct 31 03:26:58.411749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411714    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c6b06a-753f-4922-bb7c-d66063c5a790" containerName="revoke-keys"
Oct 31 03:26:58.411859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411835    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b89131-2728-4288-ac7b-0c887f6a8a7e" containerName="revoke-keys"
Oct 31 03:26:58.411859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.411844    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c6b06a-753f-4922-bb7c-d66063c5a790" containerName="revoke-keys"
Oct 31 03:26:58.416442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.416422    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.436515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.436487    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.436662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.436563    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8pqn\" (UniqueName: \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.436706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.436679    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.468609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.468580    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"]
Oct 31 03:26:58.509195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.509164    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:26:58.509393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.509292    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:26:58.538021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.537985    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.538204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.538045    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8pqn\" (UniqueName: \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.538204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.538120    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.540736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.540712    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.555130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.555101    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:26:58.555256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.555155    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:26:58.558652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.558632    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.591586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.591556    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8pqn\" (UniqueName: \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\") pod \"maas-api-revoke-sub-6a17f3c1-96pt4\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:58.730514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:58.730483    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:26:59.122005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:59.121971    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"]
Oct 31 03:26:59.122489 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:26:59.122462    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12023e58_dc6a_407b_9a3d_50b07bc4f7f2.slice/crio-009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2 WatchSource:0}: Error finding container 009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2: Status 404 returned error can't find the container with id 009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2
Oct 31 03:26:59.670323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:59.670276    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4" event={"ID":"12023e58-dc6a-407b-9a3d-50b07bc4f7f2","Type":"ContainerStarted","Data":"833dfea4c40a3a087d69fcc6efd7d5902cc77db911da0adfe3ea786ccc330f89"}
Oct 31 03:26:59.670832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:59.670339    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4" event={"ID":"12023e58-dc6a-407b-9a3d-50b07bc4f7f2","Type":"ContainerStarted","Data":"009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2"}
Oct 31 03:26:59.779782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:26:59.779662    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4" podStartSLOduration=1.779647827 podStartE2EDuration="1.779647827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:59.777407818 +0000 UTC m=+1169.791969528" watchObservedRunningTime="2026-10-31 03:26:59.779647827 +0000 UTC m=+1169.794209535"
Oct 31 03:27:00.680050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:00.680012    5831 generic.go:356] "Generic (PLEG): container finished" podID="12023e58-dc6a-407b-9a3d-50b07bc4f7f2" containerID="833dfea4c40a3a087d69fcc6efd7d5902cc77db911da0adfe3ea786ccc330f89" exitCode=0
Oct 31 03:27:00.680595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:00.680100    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4" event={"ID":"12023e58-dc6a-407b-9a3d-50b07bc4f7f2","Type":"ContainerDied","Data":"833dfea4c40a3a087d69fcc6efd7d5902cc77db911da0adfe3ea786ccc330f89"}
Oct 31 03:27:01.548347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.548318    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:27:01.552971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.552947    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584348    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584468    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584491    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584624    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584676    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584758    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584862    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2l6r\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584887    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.584956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.584944    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.686810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.686773    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.686874    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2l6r\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.686904    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.686939    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687046    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687133    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687155    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687214    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687250    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687304    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687341    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687694    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687792    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.687966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.687940    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.690371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.690346    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.690776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.690504    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.745075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.742922    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2l6r\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.747064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.745437    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.748082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.748053    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:27:01.866993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.866967    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:01.877668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.877639    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:27:01.994558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.992385    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\") pod \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") "
Oct 31 03:27:01.994558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.992510    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\") pod \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") "
Oct 31 03:27:01.994558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.992641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\" (UniqueName: \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\") pod \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\" (UID: \"12023e58-dc6a-407b-9a3d-50b07bc4f7f2\") "
Oct 31 03:27:01.994558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:01.993963    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca" pod "12023e58-dc6a-407b-9a3d-50b07bc4f7f2" (UID: "12023e58-dc6a-407b-9a3d-50b07bc4f7f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:02.003003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.002771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn" pod "12023e58-dc6a-407b-9a3d-50b07bc4f7f2" (UID: "12023e58-dc6a-407b-9a3d-50b07bc4f7f2"). InnerVolumeSpecName "kube-api-access-x8pqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:02.004893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.004862    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca" pod "12023e58-dc6a-407b-9a3d-50b07bc4f7f2" (UID: "12023e58-dc6a-407b-9a3d-50b07bc4f7f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:02.094718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.094680    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:02.094718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.094715    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:02.094718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.094725    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8pqn\" (UniqueName: \"kubernetes.io/projected/12023e58-dc6a-407b-9a3d-50b07bc4f7f2-kube-api-access-x8pqn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:02.302443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.302419    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:27:02.305778 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:02.305748    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be5abeb_6295_459f_94ed_6ad3da74b5b8.slice/crio-f8f49e0fef800d85893ef9bf665856aa01377dd2269ab8a9daba1fb093c5081c WatchSource:0}: Error finding container f8f49e0fef800d85893ef9bf665856aa01377dd2269ab8a9daba1fb093c5081c: Status 404 returned error can't find the container with id f8f49e0fef800d85893ef9bf665856aa01377dd2269ab8a9daba1fb093c5081c
Oct 31 03:27:02.308776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.308734    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:02.308879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.308862    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:02.308928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.308912    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:02.701126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.701096    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"
Oct 31 03:27:02.701628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.701095    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4" event={"ID":"12023e58-dc6a-407b-9a3d-50b07bc4f7f2","Type":"ContainerDied","Data":"009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2"}
Oct 31 03:27:02.701628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.701283    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009f547b4e277c6caf2b2f6ab86fd33f3bd9dd1c32c17a1ed9b603809dbe68b2"
Oct 31 03:27:02.703378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.703346    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" event={"ID":"4be5abeb-6295-459f-94ed-6ad3da74b5b8","Type":"ContainerStarted","Data":"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"}
Oct 31 03:27:02.703520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.703399    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" event={"ID":"4be5abeb-6295-459f-94ed-6ad3da74b5b8","Type":"ContainerStarted","Data":"f8f49e0fef800d85893ef9bf665856aa01377dd2269ab8a9daba1fb093c5081c"}
Oct 31 03:27:02.868784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.867052    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podStartSLOduration=1.8670162719999999 podStartE2EDuration="1.867016272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:02.865294152 +0000 UTC m=+1172.879855864" watchObservedRunningTime="2026-10-31 03:27:02.867016272 +0000 UTC m=+1172.881577981"
Oct 31 03:27:02.868784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:02.867127    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:03.870688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:03.870651    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:03.871078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:03.870704    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:04.283450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.280114    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"]
Oct 31 03:27:04.283450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.281123    5831 cpu_manager.go:414] "containerMap: removing container" podUID="12023e58-dc6a-407b-9a3d-50b07bc4f7f2" containerName="revoke-keys"
Oct 31 03:27:04.283450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.281144    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12023e58-dc6a-407b-9a3d-50b07bc4f7f2" containerName="revoke-keys"
Oct 31 03:27:04.283450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.281341    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="12023e58-dc6a-407b-9a3d-50b07bc4f7f2" containerName="revoke-keys"
Oct 31 03:27:04.308562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.302003    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5t4\" (UniqueName: \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472272    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472402    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472746    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472835    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.472941    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:04.483941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.473054    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:27:04.599612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.574512    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5t4\" (UniqueName: \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.599957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.599884    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.600297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.600273    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:04.697553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:04.635578    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:05.072557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.043263    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5t4\" (UniqueName: \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:05.199030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.194682    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:05.199030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.194745    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:05.249565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.248036    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:05.288665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.288604    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"]
Oct 31 03:27:05.404792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.404683    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3b8701-cbw2m\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:05.482171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.482126    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:05.870138 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.870085    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:05.870587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.870425    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:05.876378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:05.875614    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:06.337721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:06.303950    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:06.516498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:06.516412    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:06.900056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:06.891975    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:07.071185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.070112    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:07.075957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.075780    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:07.100363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.100279    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"]
Oct 31 03:27:07.101161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.101135    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:07.101400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.101376    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:07.154704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.154672    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:07.462627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.462573    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:27:07.475629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.473673    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.563488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.563436    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:27:07.571079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.571042    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m" event={"ID":"c14e2c2f-f733-4544-9b00-b8c97899245d","Type":"ContainerStarted","Data":"7722a8dff2494a4baaf436ebe5f7478698e96a5a413d82b2fe5474af8b775700"}
Oct 31 03:27:07.576716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576688    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.576867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576734    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.576867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576763    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bv7\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.576867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576800    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.576867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576857    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.577073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576900    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.577073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576935    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.577073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.576982    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.577234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.577052    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.677999    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678078    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678243    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678276    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bv7\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678316    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678405    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678417    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678455    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678499    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678575    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678834 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678603    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.678885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.678849    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.703318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.703282    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.711345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.711316    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.712981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.712954    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.731589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.731565    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.732790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.732762    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bv7\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.866198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.866163    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:07.867327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.867296    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:07.867484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:07.867364    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:08.335072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.334911    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:27:08.335521 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:08.335480    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163cc81a_c7ed_47fc_adc0_125405ec8113.slice/crio-325daf83c2fc5359db98307f3ff078d9084ab1d52400e9426843aa41ccc6c3d8 WatchSource:0}: Error finding container 325daf83c2fc5359db98307f3ff078d9084ab1d52400e9426843aa41ccc6c3d8: Status 404 returned error can't find the container with id 325daf83c2fc5359db98307f3ff078d9084ab1d52400e9426843aa41ccc6c3d8
Oct 31 03:27:08.338105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.338063    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:08.338207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.338126    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:08.338207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.338155    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:08.604647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.604544    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m" event={"ID":"c14e2c2f-f733-4544-9b00-b8c97899245d","Type":"ContainerStarted","Data":"3c9a7298834963829ef1b3b07b4a6820a0804324701e8625ee102f4f4ad3938c"}
Oct 31 03:27:08.615265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.615227    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" event={"ID":"163cc81a-c7ed-47fc-adc0-125405ec8113","Type":"ContainerStarted","Data":"a1ab1b1cae713581ab7bb6057f9b6565e141560fcbc31ad43427770cfd2303df"}
Oct 31 03:27:08.615570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.615521    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" event={"ID":"163cc81a-c7ed-47fc-adc0-125405ec8113","Type":"ContainerStarted","Data":"325daf83c2fc5359db98307f3ff078d9084ab1d52400e9426843aa41ccc6c3d8"}
Oct 31 03:27:08.725992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.725926    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m" podStartSLOduration=4.7259056 podStartE2EDuration="4.7259056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:08.715885582 +0000 UTC m=+1178.730447292" watchObservedRunningTime="2026-10-31 03:27:08.7259056 +0000 UTC m=+1178.740467311"
Oct 31 03:27:08.835905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.835840    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podStartSLOduration=1.835822917 podStartE2EDuration="1.835822917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:08.829885822 +0000 UTC m=+1178.844447544" watchObservedRunningTime="2026-10-31 03:27:08.835822917 +0000 UTC m=+1178.850384625"
Oct 31 03:27:08.867377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.867296    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:08.867595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.867569    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:08.867656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:08.867631    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:09.660906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.660869    5831 generic.go:356] "Generic (PLEG): container finished" podID="c14e2c2f-f733-4544-9b00-b8c97899245d" containerID="3c9a7298834963829ef1b3b07b4a6820a0804324701e8625ee102f4f4ad3938c" exitCode=0
Oct 31 03:27:09.661376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.660932    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m" event={"ID":"c14e2c2f-f733-4544-9b00-b8c97899245d","Type":"ContainerDied","Data":"3c9a7298834963829ef1b3b07b4a6820a0804324701e8625ee102f4f4ad3938c"}
Oct 31 03:27:09.867989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.867948    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.868154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.868019    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:09.868417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.868393    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.868516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:09.868432    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:10.867119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:10.866731    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.867119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:10.866800    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:10.867984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:10.867959    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.868164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:10.868135    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:27:10.968277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:10.968245    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:11.073640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.073600    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\" (UniqueName: \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\") pod \"c14e2c2f-f733-4544-9b00-b8c97899245d\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") "
Oct 31 03:27:11.073849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.073737    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\") pod \"c14e2c2f-f733-4544-9b00-b8c97899245d\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") "
Oct 31 03:27:11.073948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.073931    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\") pod \"c14e2c2f-f733-4544-9b00-b8c97899245d\" (UID: \"c14e2c2f-f733-4544-9b00-b8c97899245d\") "
Oct 31 03:27:11.074643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.074613    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca" pod "c14e2c2f-f733-4544-9b00-b8c97899245d" (UID: "c14e2c2f-f733-4544-9b00-b8c97899245d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:11.076308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.076283    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4" pod "c14e2c2f-f733-4544-9b00-b8c97899245d" (UID: "c14e2c2f-f733-4544-9b00-b8c97899245d"). InnerVolumeSpecName "kube-api-access-2c5t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:11.076742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.076722    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca" pod "c14e2c2f-f733-4544-9b00-b8c97899245d" (UID: "c14e2c2f-f733-4544-9b00-b8c97899245d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:11.176015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.175984    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:11.176015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.176011    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5t4\" (UniqueName: \"kubernetes.io/projected/c14e2c2f-f733-4544-9b00-b8c97899245d-kube-api-access-2c5t4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:11.176015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.176021    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14e2c2f-f733-4544-9b00-b8c97899245d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:11.689556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.688614    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"
Oct 31 03:27:11.689556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.689292    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m" event={"ID":"c14e2c2f-f733-4544-9b00-b8c97899245d","Type":"ContainerDied","Data":"7722a8dff2494a4baaf436ebe5f7478698e96a5a413d82b2fe5474af8b775700"}
Oct 31 03:27:11.689556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.689413    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7722a8dff2494a4baaf436ebe5f7478698e96a5a413d82b2fe5474af8b775700"
Oct 31 03:27:11.866672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.866630    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:11.866833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:11.866699    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:12.867396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:12.866867    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:12.867396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:12.866944    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:12.911557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:12.910639    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:12.911557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:12.910726    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:13.018621 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:27:13.007737    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:27:13.412649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.411844    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:13.412945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.412923    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:13.415609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.415394    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:27:13.819682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.819265    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"]
Oct 31 03:27:13.823166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.823133    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c14e2c2f-f733-4544-9b00-b8c97899245d" containerName="revoke-keys"
Oct 31 03:27:13.823166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.823164    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14e2c2f-f733-4544-9b00-b8c97899245d" containerName="revoke-keys"
Oct 31 03:27:13.823378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.823340    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14e2c2f-f733-4544-9b00-b8c97899245d" containerName="revoke-keys"
Oct 31 03:27:13.828376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.828351    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:13.856066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.856037    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"]
Oct 31 03:27:13.867031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.867001    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:13.867178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.867053    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:13.877598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.877569    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:13.878359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.877660    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:13.878359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.877801    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:27:13.878359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.877869    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:13.935270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.935228    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8s9f\" (UniqueName: \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:13.935454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.935293    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:13.935575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:13.935462    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.036942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.036904    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8s9f\" (UniqueName: \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.037134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.036959    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.037134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.037023    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.037841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.037769    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.039655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.039628    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.047684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.047663    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"]
Oct 31 03:27:14.052150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.052127    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feff57bd-4ngvz"]
Oct 31 03:27:14.109522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.109429    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8s9f\" (UniqueName: \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\") pod \"maas-api-revoke-sub-0069df22-bpmvs\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.136378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.136347    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e100240e-a4fa-4a78-99eb-52e94e51c954" path="/var/lib/kubelet/pods/e100240e-a4fa-4a78-99eb-52e94e51c954/volumes"
Oct 31 03:27:14.144061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.144032    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:14.560787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.560751    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"]
Oct 31 03:27:14.568961 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:14.568928    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8006bb6_9585_4e5d_a22f_30378af0645e.slice/crio-a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25 WatchSource:0}: Error finding container a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25: Status 404 returned error can't find the container with id a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25
Oct 31 03:27:14.730792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.730750    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs" event={"ID":"c8006bb6-9585-4e5d-a22f-30378af0645e","Type":"ContainerStarted","Data":"a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25"}
Oct 31 03:27:14.866896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.866859    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 03:27:14.867117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:14.866930    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 03:27:15.049922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.049886    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"]
Oct 31 03:27:15.089121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.089036    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74021052-p474x"]
Oct 31 03:27:15.093238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.093211    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"]
Oct 31 03:27:15.097638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.097622    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.155728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.155690    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"]
Oct 31 03:27:15.258539 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.258493    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.258725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.258561    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.258725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.258645    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdcb\" (UniqueName: \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.360171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.360078    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.360171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.360152    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.360407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.360212    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdcb\" (UniqueName: \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.361260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.361232    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.363420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.363393    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.416713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.416685    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdcb\" (UniqueName: \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\") pod \"maas-api-revoke-sub-59b5c7a4-g5gsc\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.719309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.719187    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:15.759292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.759246    5831 generic.go:356] "Generic (PLEG): container finished" podID="c8006bb6-9585-4e5d-a22f-30378af0645e" containerID="dbccf2baa805f6b077d406863c2ee384074dba96bdfa1670df993679aef96a8a" exitCode=0
Oct 31 03:27:15.759449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:15.759307    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs" event={"ID":"c8006bb6-9585-4e5d-a22f-30378af0645e","Type":"ContainerDied","Data":"dbccf2baa805f6b077d406863c2ee384074dba96bdfa1670df993679aef96a8a"}
Oct 31 03:27:16.145122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.145080    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77081a3b-8a48-49bf-b22d-985044fc3083" path="/var/lib/kubelet/pods/77081a3b-8a48-49bf-b22d-985044fc3083/volumes"
Oct 31 03:27:16.145862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.145838    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"]
Oct 31 03:27:16.180484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.180457    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:16.181390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.181369    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:16.183318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.183288    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:27:16.771795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.771702    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc" event={"ID":"12945d1f-09a7-434f-ac04-b9063b561410","Type":"ContainerStarted","Data":"29b6dfdbe195481988df5f7ac44355b904bd69c5e71968cac6f187eca8397fb7"}
Oct 31 03:27:16.771795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.771767    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc" event={"ID":"12945d1f-09a7-434f-ac04-b9063b561410","Type":"ContainerStarted","Data":"36561e3cd9ac406dde3ee5e902f03a5fba1897221e667efd2d1736634d31cf2d"}
Oct 31 03:27:16.946985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:16.946953    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:17.056730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.056620    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc" podStartSLOduration=2.056603093 podStartE2EDuration="2.056603093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:16.881040786 +0000 UTC m=+1186.895602495" watchObservedRunningTime="2026-10-31 03:27:17.056603093 +0000 UTC m=+1187.071164803"
Oct 31 03:27:17.093486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.093448    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\") pod \"c8006bb6-9585-4e5d-a22f-30378af0645e\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") "
Oct 31 03:27:17.093850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.093823    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\" (UniqueName: \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\") pod \"c8006bb6-9585-4e5d-a22f-30378af0645e\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") "
Oct 31 03:27:17.093976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.093947    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\") pod \"c8006bb6-9585-4e5d-a22f-30378af0645e\" (UID: \"c8006bb6-9585-4e5d-a22f-30378af0645e\") "
Oct 31 03:27:17.094372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.094341    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca" pod "c8006bb6-9585-4e5d-a22f-30378af0645e" (UID: "c8006bb6-9585-4e5d-a22f-30378af0645e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:17.094700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.094616    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:17.096049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.096025    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f" pod "c8006bb6-9585-4e5d-a22f-30378af0645e" (UID: "c8006bb6-9585-4e5d-a22f-30378af0645e"). InnerVolumeSpecName "kube-api-access-f8s9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:17.096212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.096192    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca" pod "c8006bb6-9585-4e5d-a22f-30378af0645e" (UID: "c8006bb6-9585-4e5d-a22f-30378af0645e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:17.196165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.196133    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8s9f\" (UniqueName: \"kubernetes.io/projected/c8006bb6-9585-4e5d-a22f-30378af0645e-kube-api-access-f8s9f\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:17.196165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.196168    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8006bb6-9585-4e5d-a22f-30378af0645e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:17.782175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.782137    5831 generic.go:356] "Generic (PLEG): container finished" podID="12945d1f-09a7-434f-ac04-b9063b561410" containerID="29b6dfdbe195481988df5f7ac44355b904bd69c5e71968cac6f187eca8397fb7" exitCode=0
Oct 31 03:27:17.782396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.782231    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc" event={"ID":"12945d1f-09a7-434f-ac04-b9063b561410","Type":"ContainerDied","Data":"29b6dfdbe195481988df5f7ac44355b904bd69c5e71968cac6f187eca8397fb7"}
Oct 31 03:27:17.783958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.783926    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"
Oct 31 03:27:17.783958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.783938    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs" event={"ID":"c8006bb6-9585-4e5d-a22f-30378af0645e","Type":"ContainerDied","Data":"a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25"}
Oct 31 03:27:17.784157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:17.783986    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88ea8622379018ddaa97f822585fe6be6cffc8f1b5f31c367b01903bf053f25"
Oct 31 03:27:18.974839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:18.974807    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:19.016457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.016426    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\" (UniqueName: \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\") pod \"12945d1f-09a7-434f-ac04-b9063b561410\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") "
Oct 31 03:27:19.016636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.016565    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\") pod \"12945d1f-09a7-434f-ac04-b9063b561410\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") "
Oct 31 03:27:19.016636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.016594    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\") pod \"12945d1f-09a7-434f-ac04-b9063b561410\" (UID: \"12945d1f-09a7-434f-ac04-b9063b561410\") "
Oct 31 03:27:19.017017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.016989    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca" pod "12945d1f-09a7-434f-ac04-b9063b561410" (UID: "12945d1f-09a7-434f-ac04-b9063b561410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:19.018812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.018783    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca" pod "12945d1f-09a7-434f-ac04-b9063b561410" (UID: "12945d1f-09a7-434f-ac04-b9063b561410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:19.018925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.018816    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb" pod "12945d1f-09a7-434f-ac04-b9063b561410" (UID: "12945d1f-09a7-434f-ac04-b9063b561410"). InnerVolumeSpecName "kube-api-access-6sdcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:19.118342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.118266    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:19.118342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.118295    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12945d1f-09a7-434f-ac04-b9063b561410-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:19.118342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.118306    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdcb\" (UniqueName: \"kubernetes.io/projected/12945d1f-09a7-434f-ac04-b9063b561410-kube-api-access-6sdcb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:19.802075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.802045    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"
Oct 31 03:27:19.802351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.802082    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc" event={"ID":"12945d1f-09a7-434f-ac04-b9063b561410","Type":"ContainerDied","Data":"36561e3cd9ac406dde3ee5e902f03a5fba1897221e667efd2d1736634d31cf2d"}
Oct 31 03:27:19.802351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:19.802138    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36561e3cd9ac406dde3ee5e902f03a5fba1897221e667efd2d1736634d31cf2d"
Oct 31 03:27:20.397712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.397676    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"]
Oct 31 03:27:20.398415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398396    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c8006bb6-9585-4e5d-a22f-30378af0645e" containerName="revoke-keys"
Oct 31 03:27:20.398471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398417    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8006bb6-9585-4e5d-a22f-30378af0645e" containerName="revoke-keys"
Oct 31 03:27:20.398471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398439    5831 cpu_manager.go:414] "containerMap: removing container" podUID="12945d1f-09a7-434f-ac04-b9063b561410" containerName="revoke-keys"
Oct 31 03:27:20.398471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398444    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12945d1f-09a7-434f-ac04-b9063b561410" containerName="revoke-keys"
Oct 31 03:27:20.398595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398548    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="12945d1f-09a7-434f-ac04-b9063b561410" containerName="revoke-keys"
Oct 31 03:27:20.398595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.398565    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8006bb6-9585-4e5d-a22f-30378af0645e" containerName="revoke-keys"
Oct 31 03:27:20.402513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.402497    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.437048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.437016    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28pg\" (UniqueName: \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.437217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.437150    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.437217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.437203    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.456267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.451568    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:20.456267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.451687    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:20.456267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.455647    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:20.457129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.457092    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"]
Oct 31 03:27:20.495959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.495914    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:27:20.539026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.538995    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28pg\" (UniqueName: \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.539194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.539097    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.539194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.539151    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.539982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.539962    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.541713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.541695    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.613479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.613443    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28pg\" (UniqueName: \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\") pod \"maas-api-revoke-sub-539ce198-zvthb\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:20.718123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:20.718092    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:21.099162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:21.099126    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"]
Oct 31 03:27:21.099315 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:21.099292    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef423a2_bb90_4755_b816_ee14b9d953c6.slice/crio-37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325 WatchSource:0}: Error finding container 37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325: Status 404 returned error can't find the container with id 37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325
Oct 31 03:27:21.823784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:21.823737    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb" event={"ID":"5ef423a2-bb90-4755-b816-ee14b9d953c6","Type":"ContainerStarted","Data":"d829a2b4611545d0f6bd8e5d9f9137610d82e4497e8325e584d445ff2761ba97"}
Oct 31 03:27:21.824238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:21.823801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb" event={"ID":"5ef423a2-bb90-4755-b816-ee14b9d953c6","Type":"ContainerStarted","Data":"37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325"}
Oct 31 03:27:21.935308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:21.935253    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb" podStartSLOduration=1.935238414 podStartE2EDuration="1.935238414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:21.931590355 +0000 UTC m=+1191.946152063" watchObservedRunningTime="2026-10-31 03:27:21.935238414 +0000 UTC m=+1191.949800159"
Oct 31 03:27:22.522982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.522943    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:27:22.528693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.528669    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.565939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.565907    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.565969    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566016    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566181 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566092    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566277    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566316    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566389    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnt78\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566488    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.566560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.566507    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.614955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.614923    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:27:22.667688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667647    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.667688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667687    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.667964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667724    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.667964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667755    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.667964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667874    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.667964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667903    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.667973    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnt78\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668082    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668107    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668181    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668490    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668563    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668679    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.668749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.668656    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.670610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.670590    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.670690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.670641    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.721419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.721391    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.721641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.721561    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnt78\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:22.840093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.839994    5831 generic.go:356] "Generic (PLEG): container finished" podID="5ef423a2-bb90-4755-b816-ee14b9d953c6" containerID="d829a2b4611545d0f6bd8e5d9f9137610d82e4497e8325e584d445ff2761ba97" exitCode=0
Oct 31 03:27:22.840584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.840213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb" event={"ID":"5ef423a2-bb90-4755-b816-ee14b9d953c6","Type":"ContainerDied","Data":"d829a2b4611545d0f6bd8e5d9f9137610d82e4497e8325e584d445ff2761ba97"}
Oct 31 03:27:22.843186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:22.843162    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:23.278980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.278949    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:27:23.279144 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:23.279105    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148e5fba_3872_40e2_9922_ea8e1de83ad4.slice/crio-5928b8036f1945affb4bb6e90c1bd76c9c2a1b54ea8e128d316a8a2f53ec3953 WatchSource:0}: Error finding container 5928b8036f1945affb4bb6e90c1bd76c9c2a1b54ea8e128d316a8a2f53ec3953: Status 404 returned error can't find the container with id 5928b8036f1945affb4bb6e90c1bd76c9c2a1b54ea8e128d316a8a2f53ec3953
Oct 31 03:27:23.282137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.282106    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:23.282227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.282175    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:23.282227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.282206    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:23.853311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.853273    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" event={"ID":"148e5fba-3872-40e2-9922-ea8e1de83ad4","Type":"ContainerStarted","Data":"c2f4ea80085703cd0697b2703a01abbdcd9dfc9877e0a1c6d0b0415b49c191ad"}
Oct 31 03:27:23.853844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.853357    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" event={"ID":"148e5fba-3872-40e2-9922-ea8e1de83ad4","Type":"ContainerStarted","Data":"5928b8036f1945affb4bb6e90c1bd76c9c2a1b54ea8e128d316a8a2f53ec3953"}
Oct 31 03:27:23.964344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:23.964294    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podStartSLOduration=1.964279122 podStartE2EDuration="1.964279122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:23.963872985 +0000 UTC m=+1193.978434715" watchObservedRunningTime="2026-10-31 03:27:23.964279122 +0000 UTC m=+1193.978840830"
Oct 31 03:27:24.043593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.043561    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:24.049491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.049462    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"]
Oct 31 03:27:24.055709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.055685    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c1656d-vgcxp"]
Oct 31 03:27:24.097553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.087167    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\") pod \"5ef423a2-bb90-4755-b816-ee14b9d953c6\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") "
Oct 31 03:27:24.097553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.087273    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\" (UniqueName: \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\") pod \"5ef423a2-bb90-4755-b816-ee14b9d953c6\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") "
Oct 31 03:27:24.097553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.087363    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\") pod \"5ef423a2-bb90-4755-b816-ee14b9d953c6\" (UID: \"5ef423a2-bb90-4755-b816-ee14b9d953c6\") "
Oct 31 03:27:24.097553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.088409    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca" pod "5ef423a2-bb90-4755-b816-ee14b9d953c6" (UID: "5ef423a2-bb90-4755-b816-ee14b9d953c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:24.116918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.116827    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca" pod "5ef423a2-bb90-4755-b816-ee14b9d953c6" (UID: "5ef423a2-bb90-4755-b816-ee14b9d953c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:24.126049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.126001    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg" pod "5ef423a2-bb90-4755-b816-ee14b9d953c6" (UID: "5ef423a2-bb90-4755-b816-ee14b9d953c6"). InnerVolumeSpecName "kube-api-access-x28pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:24.139441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.139412    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d077e2-f93c-4ec6-9aa8-3340c08c804b" path="/var/lib/kubelet/pods/52d077e2-f93c-4ec6-9aa8-3340c08c804b/volumes"
Oct 31 03:27:24.219629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.192842    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:24.219842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.219682    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28pg\" (UniqueName: \"kubernetes.io/projected/5ef423a2-bb90-4755-b816-ee14b9d953c6-kube-api-access-x28pg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:24.219842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.219714    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef423a2-bb90-4755-b816-ee14b9d953c6-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:24.844247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.844211    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:24.845723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.845703    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:24.845827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.845745    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:24.862072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.862047    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"
Oct 31 03:27:24.862417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.862086    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb" event={"ID":"5ef423a2-bb90-4755-b816-ee14b9d953c6","Type":"ContainerDied","Data":"37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325"}
Oct 31 03:27:24.862417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:24.862135    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cdb4a177317f587230f1b596d3f87cd4a75a0d55320bdbdcfb5c4be0491325"
Oct 31 03:27:25.351582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:25.351551    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:27:25.852579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:25.844274    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:25.852579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:25.844338    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:26.861613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:26.856997    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:26.861613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:26.857113    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:27.580747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:27.580685    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:27.844086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:27.843986    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:27.844353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:27.844321    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:27.905690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:27.905632    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.175784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.174467    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.845554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.844949    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:28.845554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.845030    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:28.889567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.888648    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:28.889567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.888718    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.920560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.918446    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.920560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:28.918910    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:29.188849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.187472    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 31 03:27:29.188849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.187968    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:29.560729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.560631    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:29.586556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.560966    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.851652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.821004    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:29.851652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.821063    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.919716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.919673    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 31 03:27:29.931068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.921628    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:29.931068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.921710    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.931068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.922621    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:29.931068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:29.922662    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:30.254602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:30.254556    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.263559 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:27:31.238513    5831 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.014s"
Oct 31 03:27:31.507557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.504212    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"]
Oct 31 03:27:31.507557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.412431    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:31.507557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.505020    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:31.543324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.543215    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.916447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.916307    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:31.916789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:31.916757    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:32.020915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.020886    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:32.056400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.056368    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:32.091701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.085775    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8013e78-96xsf"]
Oct 31 03:27:32.318698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.318657    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:32.392729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.330619    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.475594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.475558    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:32.475959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.475926    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.715568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.715510    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5796bd11-b982-4c62-a657-a9b7fab3d829" path="/var/lib/kubelet/pods/5796bd11-b982-4c62-a657-a9b7fab3d829/volumes"
Oct 31 03:27:32.716280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.716257    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:27:32.717981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.717960    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5ef423a2-bb90-4755-b816-ee14b9d953c6" containerName="revoke-keys"
Oct 31 03:27:32.718121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.718106    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef423a2-bb90-4755-b816-ee14b9d953c6" containerName="revoke-keys"
Oct 31 03:27:32.718459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.718439    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef423a2-bb90-4755-b816-ee14b9d953c6" containerName="revoke-keys"
Oct 31 03:27:32.761570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.750034    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:32.761570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.750113    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.926840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.907985    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:32.927241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:32.927202    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:33.026905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.019794    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:27:33.042100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.027416    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.086745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.086700    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.131909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.115499    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b3a29ec-openshift-default-dockercfg-4ddsz\""
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.233787    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.233869    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.233924    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234070    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234096    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234124    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234191    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jl9\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.250561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.234337    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.274640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.266443    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.399846    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.399949    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400049    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400123    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400346    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400376    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400407    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.400603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.400518    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jl9\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.416376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.416338    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.464649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.417698    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.464941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.417948    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.465043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.418374    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.465138 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.455942    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.487566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.475127    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.531555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.502166    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.531555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.526426    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jl9\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.573771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.572768    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.650263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.650229    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn\" (UID: \"02690735-c044-4790-be99-54e672e02286\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:33.769702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.769643    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:33.773558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.769969    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.850560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.844157    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:33.907645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.907602    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:33.955582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:33.955437    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:34.194567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.155889    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:34.400748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.250978    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:34.559891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.559842    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:34.560193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.278065    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:34.583600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.541015    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:34.583600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.576518    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:34.798325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.798293    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zp2v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:34.798547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.798502    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:34.808573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.805063    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:34.816676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.816645    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:34.871685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.853086    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:34.871685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.853139    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:34.871685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:34.853233    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:35.420178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:35.420122    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:35.733133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:35.733103    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:27:36.058942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:36.058839    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:36.058942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:36.058913    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:36.077888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:36.060790    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:36.081054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:36.081022    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:36.159192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:36.148980    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:27:36.762759 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:36.759653    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02690735_c044_4790_be99_54e672e02286.slice/crio-8fca904427705a644dfc0546cb4a1771bb17dd7df84c5ad7a614ce3356b113f2 WatchSource:0}: Error finding container 8fca904427705a644dfc0546cb4a1771bb17dd7df84c5ad7a614ce3356b113f2: Status 404 returned error can't find the container with id 8fca904427705a644dfc0546cb4a1771bb17dd7df84c5ad7a614ce3356b113f2
Oct 31 03:27:37.101406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.101368    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:37.101692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.101658    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:37.319564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.311450    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:37.319564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.311635    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:37.319564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.311773    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:37.362375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.351273    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerStarted","Data":"8fca904427705a644dfc0546cb4a1771bb17dd7df84c5ad7a614ce3356b113f2"}
Oct 31 03:27:37.422555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.395470    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:27:37.646602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.631479    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:27:37.774697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.774632    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:37.802306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.802271    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:27:37.868592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.849356    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:37.868592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.850196    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:37.868592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.850341    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:37.903612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.893562    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:27:37.937050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:37.937016    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:27:38.376574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:38.372477    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerStarted","Data":"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"}
Oct 31 03:27:38.536686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:38.536625    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podStartSLOduration=8.536603367 podStartE2EDuration="8.536603367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:38.532104596 +0000 UTC m=+1208.546666331" watchObservedRunningTime="2026-10-31 03:27:38.536603367 +0000 UTC m=+1208.551165075"
Oct 31 03:27:38.843684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:38.843647    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 03:27:38.844073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:38.843714    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 03:27:38.956136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:38.956095    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:39.964260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:39.964224    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:40.064949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.064915    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"]
Oct 31 03:27:40.069591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.069557    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa76307-h5k6z"]
Oct 31 03:27:40.140170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.140114    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436ec380-7dcb-43d5-a75b-70b7108041bc" path="/var/lib/kubelet/pods/436ec380-7dcb-43d5-a75b-70b7108041bc/volumes"
Oct 31 03:27:40.396348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.396249    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:40.397738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.397715    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:27:40.886368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.885710    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:40.886368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:40.885796    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:41.247906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:41.247868    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:41.249017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:41.248995    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:27:42.983752 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:27:42.983716    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:27:45.099452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.091036    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"]
Oct 31 03:27:45.227345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.227306    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16d4902-22jfb"]
Oct 31 03:27:45.406674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.406588    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:27:45.757555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.754759    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:27:45.757555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.754888    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:45.824657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.824625    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b3a29ec-dockercfg-fw6vz\""
Oct 31 03:27:45.853777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.851993    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6d2\" (UniqueName: \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:45.853777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.852137    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:45.870592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.870560    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b3a29ec\""
Oct 31 03:27:45.953638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.953602    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:45.953821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.953741    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6d2\" (UniqueName: \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:45.954322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:45.954297    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:46.008514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:46.008438    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6d2\" (UniqueName: \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:46.112294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:46.112263    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:46.140470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:46.140438    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95792043-3c91-452d-baab-438f2de0fedf" path="/var/lib/kubelet/pods/95792043-3c91-452d-baab-438f2de0fedf/volumes"
Oct 31 03:27:46.516672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:46.516637    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:27:46.641723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:46.641626    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" event={"ID":"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30","Type":"ContainerStarted","Data":"b38bd4a64c17e9a1b0fa6ad5521eeb62a1139a01022a71a068f2fe46715e528a"}
Oct 31 03:27:47.654180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:47.654137    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" event={"ID":"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30","Type":"ContainerStarted","Data":"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"}
Oct 31 03:27:47.654605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:47.654228    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:27:47.764801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:47.764746    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" podStartSLOduration=2.764731549 podStartE2EDuration="2.764731549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:47.76242218 +0000 UTC m=+1217.776983888" watchObservedRunningTime="2026-10-31 03:27:47.764731549 +0000 UTC m=+1217.779293433"
Oct 31 03:27:52.536360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.536317    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:27:52.541396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.541370    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564673    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564741    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564771    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564829    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564860    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564891    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564918    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.564976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.564957    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d9d\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.651500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.651466    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:27:52.666801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666758    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666843    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666876    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666912    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666950    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.666996    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.667036    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.667093    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d9d\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.667166    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667679 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.667640    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.667946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.667919    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.668255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.668219    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.668727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.668703    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.670176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.670130    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.672418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.672393    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.675244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.675214    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.769554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.760273    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.792003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.783331    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d9d\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:52.872078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:52.872041    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:53.563417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:53.563379    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:27:53.575134 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:27:53.574794    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c25628_063f_4996_b147_8210e12ce7fb.slice/crio-4832af33a73c971d40ecfe5f986983b4ee92f4de1912b1dcb693d6b03175541b WatchSource:0}: Error finding container 4832af33a73c971d40ecfe5f986983b4ee92f4de1912b1dcb693d6b03175541b: Status 404 returned error can't find the container with id 4832af33a73c971d40ecfe5f986983b4ee92f4de1912b1dcb693d6b03175541b
Oct 31 03:27:53.581776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:53.581622    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:53.582194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:53.582165    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:53.582579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:53.582462    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:27:53.786572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:53.784750    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" event={"ID":"51c25628-063f-4996-b147-8210e12ce7fb","Type":"ContainerStarted","Data":"4832af33a73c971d40ecfe5f986983b4ee92f4de1912b1dcb693d6b03175541b"}
Oct 31 03:27:54.926555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:54.926416    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" event={"ID":"51c25628-063f-4996-b147-8210e12ce7fb","Type":"ContainerStarted","Data":"19bf18d93e2405f22a6527ba43b4ee3239f51565e6570af85922aa0b8bb09c8e"}
Oct 31 03:27:55.098555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:55.094698    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.136682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:55.136641    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"]
Oct 31 03:27:55.221556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:55.204415    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podStartSLOduration=3.204394906 podStartE2EDuration="3.204394906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:55.137131866 +0000 UTC m=+1225.151693600" watchObservedRunningTime="2026-10-31 03:27:55.204394906 +0000 UTC m=+1225.218956616"
Oct 31 03:27:55.221556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:55.204573    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cedb47-mgxft"]
Oct 31 03:27:56.042555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.039398    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 31 03:27:56.111180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.109294    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:27:56.111180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.109449    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:27:56.111180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.109495    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:27:56.111180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.109591    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.111180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.109971    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:27:56.633555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:56.591093    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4239c725-9830-4494-b4b2-2f6a6b50af58" path="/var/lib/kubelet/pods/4239c725-9830-4494-b4b2-2f6a6b50af58/volumes"
Oct 31 03:27:57.008556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.007765    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:27:57.008556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.007863    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:27:57.063578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.044696    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.063578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.045116    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:27:57.145029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.144986    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 31 03:27:57.339485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.339389    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:57.361886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.361842    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.607369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.597640    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 31 03:27:57.922279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.902395    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:27:57.922279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:57.920636    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:27:58.082808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:58.080553    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:58.181687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:58.156720    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:59.078691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.078626    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:27:59.091715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.078972    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:27:59.112757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.112607    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:59.217555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.214327    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:59.372877 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:27:59.372780    5831 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.236s"
Oct 31 03:27:59.373186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.373168    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:28:00.033556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.780843    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 03:28:00.033556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.942257    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:00.033556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:27:59.942310    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:00.208555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:00.189099    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 31 03:28:00.773678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:00.773629    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:28:00.773978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:00.773942    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:28:00.887189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:00.887153    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:00.887473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:00.887442    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:01.164615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:01.159042    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 31 03:28:01.403785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:01.403738    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:28:01.466777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:01.466716    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 03:28:01.873544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:01.873451    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:01.873544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:01.873502    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:02.121062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.121030    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"]
Oct 31 03:28:02.127352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.127285    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.154811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.154778    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.154979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.154885    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.154979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.154938    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscj9\" (UniqueName: \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.175895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.175863    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:02.175895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.175891    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:02.176134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.175969    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:28:02.176356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.176332    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:02.192813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.192774    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"]
Oct 31 03:28:02.256708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.256678    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscj9\" (UniqueName: \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.256879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.256863    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.256925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.256908    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.257590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.257567    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.259397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.259377    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.310688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.310659    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscj9\" (UniqueName: \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\") pod \"maas-api-revoke-sub-b35eea44-rt2nv\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.443843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.443805    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:02.845987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.845957    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"]
Oct 31 03:28:02.846244 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:28:02.846217    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614c321e_1034_4c7a_a78b_3f09183b05b2.slice/crio-0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b WatchSource:0}: Error finding container 0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b: Status 404 returned error can't find the container with id 0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b
Oct 31 03:28:02.872869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.872842    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:02.872977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:02.872892    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:03.566469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:03.566380    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv" event={"ID":"614c321e-1034-4c7a-a78b-3f09183b05b2","Type":"ContainerStarted","Data":"53dae62e6b4699fa295c9dd4910ba9b9ed47a92b7af60814c64bfc5bedbc05da"}
Oct 31 03:28:03.566469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:03.566439    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv" event={"ID":"614c321e-1034-4c7a-a78b-3f09183b05b2","Type":"ContainerStarted","Data":"0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b"}
Oct 31 03:28:03.676262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:03.676199    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv" podStartSLOduration=1.676182361 podStartE2EDuration="1.676182361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:03.674583209 +0000 UTC m=+1233.689144919" watchObservedRunningTime="2026-10-31 03:28:03.676182361 +0000 UTC m=+1233.690744070"
Oct 31 03:28:03.873118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:03.873032    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:03.873118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:03.873099    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:04.577872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:04.577835    5831 generic.go:356] "Generic (PLEG): container finished" podID="614c321e-1034-4c7a-a78b-3f09183b05b2" containerID="53dae62e6b4699fa295c9dd4910ba9b9ed47a92b7af60814c64bfc5bedbc05da" exitCode=0
Oct 31 03:28:04.578325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:04.577926    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv" event={"ID":"614c321e-1034-4c7a-a78b-3f09183b05b2","Type":"ContainerDied","Data":"53dae62e6b4699fa295c9dd4910ba9b9ed47a92b7af60814c64bfc5bedbc05da"}
Oct 31 03:28:04.873167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:04.873087    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 31 03:28:04.873167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:04.873151    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 31 03:28:05.815289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.815260    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:05.941948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.941913    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\") pod \"614c321e-1034-4c7a-a78b-3f09183b05b2\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") "
Oct 31 03:28:05.942177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.942156    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\") pod \"614c321e-1034-4c7a-a78b-3f09183b05b2\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") "
Oct 31 03:28:05.942265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.942222    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\" (UniqueName: \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\") pod \"614c321e-1034-4c7a-a78b-3f09183b05b2\" (UID: \"614c321e-1034-4c7a-a78b-3f09183b05b2\") "
Oct 31 03:28:05.942705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.942675    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca" pod "614c321e-1034-4c7a-a78b-3f09183b05b2" (UID: "614c321e-1034-4c7a-a78b-3f09183b05b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:05.942928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.942908    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:05.945623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.945593    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca" pod "614c321e-1034-4c7a-a78b-3f09183b05b2" (UID: "614c321e-1034-4c7a-a78b-3f09183b05b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:05.962335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:05.962173    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9" pod "614c321e-1034-4c7a-a78b-3f09183b05b2" (UID: "614c321e-1034-4c7a-a78b-3f09183b05b2"). InnerVolumeSpecName "kube-api-access-mscj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:06.015996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.015931    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:28:06.017190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.016772    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:28:06.019788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.019767    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:28:06.044733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.044702    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mscj9\" (UniqueName: \"kubernetes.io/projected/614c321e-1034-4c7a-a78b-3f09183b05b2-kube-api-access-mscj9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:06.044733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.044730    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614c321e-1034-4c7a-a78b-3f09183b05b2-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:06.596824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.596794    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"
Oct 31 03:28:06.597007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.596825    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv" event={"ID":"614c321e-1034-4c7a-a78b-3f09183b05b2","Type":"ContainerDied","Data":"0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b"}
Oct 31 03:28:06.597007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:06.596874    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d09fad3bd28f1452462c732cbfa2a29319c1b17a539bdeeabd48d6de131a14b"
Oct 31 03:28:12.981469 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:28:12.981430    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:28:13.702793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:13.702762    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:28:16.233553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.232918    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"]
Oct 31 03:28:16.246555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.234199    5831 cpu_manager.go:414] "containerMap: removing container" podUID="614c321e-1034-4c7a-a78b-3f09183b05b2" containerName="revoke-keys"
Oct 31 03:28:16.246555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.234223    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614c321e-1034-4c7a-a78b-3f09183b05b2" containerName="revoke-keys"
Oct 31 03:28:16.246555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.234444    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="614c321e-1034-4c7a-a78b-3f09183b05b2" containerName="revoke-keys"
Oct 31 03:28:16.248800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.247997    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.338079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.338043    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"]
Oct 31 03:28:16.338898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.338870    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:28:16.339116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.339100    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:16.339348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.339332    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:16.346552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.340318    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9hp\" (UniqueName: \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.346915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.346887    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.347235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.347216    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.347711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.347690    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:16.460620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.449817    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9hp\" (UniqueName: \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.461397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.461361    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.465556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.461770    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.486073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.485994    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.593507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.593475    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.613377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.613307    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9hp\" (UniqueName: \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\") pod \"maas-api-revoke-sub-267c6524-krc5z\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:16.893605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:16.892802    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:17.777361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:17.777312    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:18.025219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.006422    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:18.048554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.038795    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.300247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.260857    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.300247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.261014    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.300247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.261176    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.300247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.261215    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmgw\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.369227    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.369282    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmgw\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.369330    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.369460    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.409444    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.438568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.437941    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.634991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.634953    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:18.924115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.924078    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"]
Oct 31 03:28:18.954458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:18.954418    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmgw\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\") pod \"payload-processing-external-model-55bb6bdfd4-xdcbf\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") " pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:19.009572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.007021    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:28:19.009572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.007108    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:28:19.203551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.198515    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:19.504728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.504684    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z" event={"ID":"23738506-c912-4b44-b007-04351abad98d","Type":"ContainerStarted","Data":"46c27379a8f5b87e289179eeb7c31e989db975ddac5e6040b9045e8d1ff7f15c"}
Oct 31 03:28:19.504915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.504758    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z" event={"ID":"23738506-c912-4b44-b007-04351abad98d","Type":"ContainerStarted","Data":"c4f682ec5c7434fc21d169e5c9df95d81a1cb2926d521c8b3d05a6b786deddd2"}
Oct 31 03:28:19.612644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.612577    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z" podStartSLOduration=3.6125209270000003 podStartE2EDuration="3.612520927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:19.612266231 +0000 UTC m=+1249.626827940" watchObservedRunningTime="2026-10-31 03:28:19.612520927 +0000 UTC m=+1249.627082636"
Oct 31 03:28:19.728703 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:28:19.727395    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c8a681_e5d6_4cd9_8c9f_d1196bf1b3da.slice/crio-440c845ea080d32a836e04297f6d7ba9621625401470017caca9cb2f01aa4b3b WatchSource:0}: Error finding container 440c845ea080d32a836e04297f6d7ba9621625401470017caca9cb2f01aa4b3b: Status 404 returned error can't find the container with id 440c845ea080d32a836e04297f6d7ba9621625401470017caca9cb2f01aa4b3b
Oct 31 03:28:19.728703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:19.727581    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:20.517829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.517793    5831 generic.go:356] "Generic (PLEG): container finished" podID="23738506-c912-4b44-b007-04351abad98d" containerID="46c27379a8f5b87e289179eeb7c31e989db975ddac5e6040b9045e8d1ff7f15c" exitCode=0
Oct 31 03:28:20.518215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.517875    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z" event={"ID":"23738506-c912-4b44-b007-04351abad98d","Type":"ContainerDied","Data":"46c27379a8f5b87e289179eeb7c31e989db975ddac5e6040b9045e8d1ff7f15c"}
Oct 31 03:28:20.519838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.519812    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" event={"ID":"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da","Type":"ContainerStarted","Data":"e096ee1f718b53b423e7d002b99c538b79180838221bfa2847b883377b5e4d50"}
Oct 31 03:28:20.519953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.519863    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" event={"ID":"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da","Type":"ContainerStarted","Data":"440c845ea080d32a836e04297f6d7ba9621625401470017caca9cb2f01aa4b3b"}
Oct 31 03:28:20.519953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.519906    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:20.735480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:20.735426    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" podStartSLOduration=3.73541076 podStartE2EDuration="3.73541076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:20.730944554 +0000 UTC m=+1250.745506263" watchObservedRunningTime="2026-10-31 03:28:20.73541076 +0000 UTC m=+1250.749972468"
Oct 31 03:28:21.707910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.707881    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:21.820779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.820674    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\") pod \"23738506-c912-4b44-b007-04351abad98d\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") "
Oct 31 03:28:21.820779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.820774    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\" (UniqueName: \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\") pod \"23738506-c912-4b44-b007-04351abad98d\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") "
Oct 31 03:28:21.821012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.820956    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\") pod \"23738506-c912-4b44-b007-04351abad98d\" (UID: \"23738506-c912-4b44-b007-04351abad98d\") "
Oct 31 03:28:21.831235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.830856    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca" pod "23738506-c912-4b44-b007-04351abad98d" (UID: "23738506-c912-4b44-b007-04351abad98d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:21.841866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.841824    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp" pod "23738506-c912-4b44-b007-04351abad98d" (UID: "23738506-c912-4b44-b007-04351abad98d"). InnerVolumeSpecName "kube-api-access-rl9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:21.842477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.842448    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca" pod "23738506-c912-4b44-b007-04351abad98d" (UID: "23738506-c912-4b44-b007-04351abad98d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:21.923292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.923255    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23738506-c912-4b44-b007-04351abad98d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:21.923292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.923288    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9hp\" (UniqueName: \"kubernetes.io/projected/23738506-c912-4b44-b007-04351abad98d-kube-api-access-rl9hp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:21.923499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:21.923302    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23738506-c912-4b44-b007-04351abad98d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:22.538934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:22.538905    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"
Oct 31 03:28:22.539179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:22.538945    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z" event={"ID":"23738506-c912-4b44-b007-04351abad98d","Type":"ContainerDied","Data":"c4f682ec5c7434fc21d169e5c9df95d81a1cb2926d521c8b3d05a6b786deddd2"}
Oct 31 03:28:22.539179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:22.539011    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f682ec5c7434fc21d169e5c9df95d81a1cb2926d521c8b3d05a6b786deddd2"
Oct 31 03:28:28.558441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:28.558189    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:28:28.578744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:28.578695    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:28.581934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:28.581908    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:32.946639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:32.946607    5831 scope.go:122] "RemoveContainer" containerID="4c6ea27b808dafceb284d9249aae9056c4a4a2729ed96403424d1086b711a38a"
Oct 31 03:28:32.963202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:32.963180    5831 scope.go:122] "RemoveContainer" containerID="8e0a30bc75c4de2463277b16d8a65e4ac58dcd6abf4c5e3aa82dfea89fe7a0b7"
Oct 31 03:28:32.973942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:32.973920    5831 scope.go:122] "RemoveContainer" containerID="640135dc13f727fced192c0c88d6c95c059f44e6221daca5db77966b84854689"
Oct 31 03:28:32.984764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:32.984740    5831 scope.go:122] "RemoveContainer" containerID="73ba75af73a32c8295b6135fa93c825622de16acc3aef255b7152c5a98f55e20"
Oct 31 03:28:32.994745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:32.994717    5831 scope.go:122] "RemoveContainer" containerID="467d2e7df09e05935056f3f3735b0147b93074a6998e17494d1052aef474bc50"
Oct 31 03:28:33.559928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.559884    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" podUID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" containerName="payload-processing" containerID="cri-o://2f60136f95174d1f779498f335379e2253b2bf016fc931b5fa2616b1c251b8ab" gracePeriod=25
Oct 31 03:28:33.579238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.579180    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" podUID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" containerName="payload-processing" containerID="cri-o://e096ee1f718b53b423e7d002b99c538b79180838221bfa2847b883377b5e4d50" gracePeriod=25
Oct 31 03:28:33.720926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.720882    5831 generic.go:356] "Generic (PLEG): container finished" podID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" containerID="e096ee1f718b53b423e7d002b99c538b79180838221bfa2847b883377b5e4d50" exitCode=0
Oct 31 03:28:33.721121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.721099    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" event={"ID":"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da","Type":"ContainerDied","Data":"e096ee1f718b53b423e7d002b99c538b79180838221bfa2847b883377b5e4d50"}
Oct 31 03:28:33.723310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.723283    5831 generic.go:356] "Generic (PLEG): container finished" podID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" containerID="2f60136f95174d1f779498f335379e2253b2bf016fc931b5fa2616b1c251b8ab" exitCode=0
Oct 31 03:28:33.723433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.723418    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" event={"ID":"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64","Type":"ContainerDied","Data":"2f60136f95174d1f779498f335379e2253b2bf016fc931b5fa2616b1c251b8ab"}
Oct 31 03:28:33.900214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.900177    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:28:33.940518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.940492    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:33.965949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.965918    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\") pod \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") "
Oct 31 03:28:33.966393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966015    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\") pod \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") "
Oct 31 03:28:33.966393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966092    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\") pod \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") "
Oct 31 03:28:33.966393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966242    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\") pod \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\" (UID: \"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64\") "
Oct 31 03:28:33.966393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966289    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume" pod "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" (UID: "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:33.966655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966481    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume" pod "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" (UID: "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:33.966766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966749    5831 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-routing-overlay-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:33.966822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.966770    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:33.968441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.968422    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials" pod "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" (UID: "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:33.968441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:33.968435    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w" pod "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" (UID: "1e2b1200-ad45-462d-ab2f-c1b9a75d1c64"). InnerVolumeSpecName "kube-api-access-wh48w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:34.067757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.067727    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\") pod \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") "
Oct 31 03:28:34.067952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.067805    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\") pod \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") "
Oct 31 03:28:34.067952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.067870    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\") pod \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") "
Oct 31 03:28:34.067952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.067939    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\") pod \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\" (UID: \"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da\") "
Oct 31 03:28:34.068240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068213    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume" pod "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" (UID: "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:34.068301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068238    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume" pod "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" (UID: "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:34.068611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068589    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.068611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068611    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh48w\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-kube-api-access-wh48w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.068815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068624    5831 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-routing-overlay-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.068815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.068634    5831 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64-provider-credentials\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.070194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.070168    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials" pod "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" (UID: "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:34.070278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.070254    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw" pod "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" (UID: "a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da"). InnerVolumeSpecName "kube-api-access-9dmgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:34.170189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.170158    5831 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-provider-credentials\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.170323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.170194    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmgw\" (UniqueName: \"kubernetes.io/projected/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da-kube-api-access-9dmgw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.734511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.734474    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"
Oct 31 03:28:34.734511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.734499    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft" event={"ID":"1e2b1200-ad45-462d-ab2f-c1b9a75d1c64","Type":"ContainerDied","Data":"4fe1e7e97e09ca7030fea6a97ded8e73361af00d1f26ac252be6c1b82c206147"}
Oct 31 03:28:34.734797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.734598    5831 scope.go:122] "RemoveContainer" containerID="2f60136f95174d1f779498f335379e2253b2bf016fc931b5fa2616b1c251b8ab"
Oct 31 03:28:34.738019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.736934    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf" event={"ID":"a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da","Type":"ContainerDied","Data":"440c845ea080d32a836e04297f6d7ba9621625401470017caca9cb2f01aa4b3b"}
Oct 31 03:28:34.738019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.737026    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"
Oct 31 03:28:34.749769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.749750    5831 scope.go:122] "RemoveContainer" containerID="e096ee1f718b53b423e7d002b99c538b79180838221bfa2847b883377b5e4d50"
Oct 31 03:28:34.896223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.896125    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:28:34.935870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:34.935831    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfbf6cc9d-sxpft"]
Oct 31 03:28:35.004178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:35.004144    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:35.009485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:35.008830    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55bb6bdfd4-xdcbf"]
Oct 31 03:28:36.137625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:36.137595    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" path="/var/lib/kubelet/pods/1e2b1200-ad45-462d-ab2f-c1b9a75d1c64/volumes"
Oct 31 03:28:36.138078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:36.138059    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" path="/var/lib/kubelet/pods/a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da/volumes"
Oct 31 03:28:43.085556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:43.083053    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:28:43.158579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:43.157647    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 31 03:28:43.400353 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:28:43.395060    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:28:43.564688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:43.564634    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:28:44.573195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.573158    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"]
Oct 31 03:28:44.575405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575376    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" containerName="payload-processing"
Oct 31 03:28:44.575405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575400    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" containerName="payload-processing"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575412    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" containerName="payload-processing"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575418    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" containerName="payload-processing"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575462    5831 cpu_manager.go:414] "containerMap: removing container" podUID="23738506-c912-4b44-b007-04351abad98d" containerName="revoke-keys"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575466    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23738506-c912-4b44-b007-04351abad98d" containerName="revoke-keys"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575643    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="23738506-c912-4b44-b007-04351abad98d" containerName="revoke-keys"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575659    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c8a681-e5d6-4cd9-8c9f-d1196bf1b3da" containerName="payload-processing"
Oct 31 03:28:44.575671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.575667    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2b1200-ad45-462d-ab2f-c1b9a75d1c64" containerName="payload-processing"
Oct 31 03:28:44.596853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.596822    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.633155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.633124    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"]
Oct 31 03:28:44.669827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.669794    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:44.670024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.669892    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:44.670024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.669901    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:28:44.701895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.701864    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:44.722758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.722723    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.723044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.723018    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.723301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.723282    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7hp\" (UniqueName: \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.825275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.825187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.825275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.825227    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.825461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.825303    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7hp\" (UniqueName: \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.826134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.826094    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.828913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.828897    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.878495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.878460    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7hp\" (UniqueName: \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\") pod \"maas-api-revoke-sub-39186f42-8b882\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:44.926290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:44.926256    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:45.331163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:45.331128    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"]
Oct 31 03:28:46.038069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:46.037972    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882" event={"ID":"235b8811-eb3c-4d47-bce6-9095e52520e0","Type":"ContainerStarted","Data":"43a3609c8688a0d16dc974c782c7ce5ff677983024254a8e8f9b14c2b152b40c"}
Oct 31 03:28:46.038069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:46.038041    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882" event={"ID":"235b8811-eb3c-4d47-bce6-9095e52520e0","Type":"ContainerStarted","Data":"174744bbac6874fa77831cc950424da08a4037bd6b70ac0c82614bcbe98dcaae"}
Oct 31 03:28:46.147366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:46.147316    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882" podStartSLOduration=2.1473018919999998 podStartE2EDuration="2.147301892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:46.145283537 +0000 UTC m=+1276.159845244" watchObservedRunningTime="2026-10-31 03:28:46.147301892 +0000 UTC m=+1276.161863599"
Oct 31 03:28:47.054262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:47.054230    5831 generic.go:356] "Generic (PLEG): container finished" podID="235b8811-eb3c-4d47-bce6-9095e52520e0" containerID="43a3609c8688a0d16dc974c782c7ce5ff677983024254a8e8f9b14c2b152b40c" exitCode=0
Oct 31 03:28:47.054726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:47.054310    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882" event={"ID":"235b8811-eb3c-4d47-bce6-9095e52520e0","Type":"ContainerDied","Data":"43a3609c8688a0d16dc974c782c7ce5ff677983024254a8e8f9b14c2b152b40c"}
Oct 31 03:28:48.265458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.265425    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:48.380769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.380688    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\" (UniqueName: \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\") pod \"235b8811-eb3c-4d47-bce6-9095e52520e0\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") "
Oct 31 03:28:48.381136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.381115    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\") pod \"235b8811-eb3c-4d47-bce6-9095e52520e0\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") "
Oct 31 03:28:48.381318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.381303    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\") pod \"235b8811-eb3c-4d47-bce6-9095e52520e0\" (UID: \"235b8811-eb3c-4d47-bce6-9095e52520e0\") "
Oct 31 03:28:48.381464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.381433    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca" pod "235b8811-eb3c-4d47-bce6-9095e52520e0" (UID: "235b8811-eb3c-4d47-bce6-9095e52520e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:48.382123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.382069    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:48.396639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.396198    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp" pod "235b8811-eb3c-4d47-bce6-9095e52520e0" (UID: "235b8811-eb3c-4d47-bce6-9095e52520e0"). InnerVolumeSpecName "kube-api-access-tn7hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:48.396639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.396334    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca" pod "235b8811-eb3c-4d47-bce6-9095e52520e0" (UID: "235b8811-eb3c-4d47-bce6-9095e52520e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:48.483974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.483940    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235b8811-eb3c-4d47-bce6-9095e52520e0-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:48.483974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:48.483971    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7hp\" (UniqueName: \"kubernetes.io/projected/235b8811-eb3c-4d47-bce6-9095e52520e0-kube-api-access-tn7hp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:49.085601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:49.083962    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882" event={"ID":"235b8811-eb3c-4d47-bce6-9095e52520e0","Type":"ContainerDied","Data":"174744bbac6874fa77831cc950424da08a4037bd6b70ac0c82614bcbe98dcaae"}
Oct 31 03:28:49.085601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:49.084066    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="174744bbac6874fa77831cc950424da08a4037bd6b70ac0c82614bcbe98dcaae"
Oct 31 03:28:49.085601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:49.084114    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"
Oct 31 03:28:52.119628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.119582    5831 generic.go:356] "Generic (PLEG): container finished" podID="02690735-c044-4790-be99-54e672e02286" containerID="82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e" exitCode=0
Oct 31 03:28:52.120096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.119654    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerDied","Data":"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"}
Oct 31 03:28:52.120096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.120083    5831 scope.go:122] "RemoveContainer" containerID="82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"
Oct 31 03:28:52.121553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.121513    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:28:52.121629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.121595    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:28:52.121679 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:52.121625    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:28:53.136523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:53.136483    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerStarted","Data":"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"}
Oct 31 03:28:53.956684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:53.956641    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:28:53.957771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:53.957742    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 03:28:53.957911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:53.957804    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 03:28:54.956897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:54.956858    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 03:28:54.957297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:54.956937    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 03:28:55.399718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.399628    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:28:55.407587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.407551    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"]
Oct 31 03:28:55.408221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.408208    5831 cpu_manager.go:414] "containerMap: removing container" podUID="235b8811-eb3c-4d47-bce6-9095e52520e0" containerName="revoke-keys"
Oct 31 03:28:55.408281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.408223    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235b8811-eb3c-4d47-bce6-9095e52520e0" containerName="revoke-keys"
Oct 31 03:28:55.408345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.408336    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="235b8811-eb3c-4d47-bce6-9095e52520e0" containerName="revoke-keys"
Oct 31 03:28:55.412386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.412354    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.447909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.447880    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"]
Oct 31 03:28:55.503556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.497053    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:55.506677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.506646    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:55.506817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.506726    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:28:55.506910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.506838    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:55.509743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.509717    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.510147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.510127    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.510343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.510324    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td26x\" (UniqueName: \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.612278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.612215    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.612479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.612311    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td26x\" (UniqueName: \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.612479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.612415    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.612980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.612955    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.615128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.615104    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.665575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.665495    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td26x\" (UniqueName: \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\") pod \"maas-api-revoke-sub-a7def1a8-w6s6c\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.726962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.726926    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:55.956506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.956468    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 03:28:55.956715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:55.956567    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 03:28:56.231614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:56.224990    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"]
Oct 31 03:28:56.955971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:56.955927    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 03:28:56.956112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:56.955991    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 03:28:57.357227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:57.357189    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c" event={"ID":"dbd298c3-0d0b-48e3-8d28-f0033803ce09","Type":"ContainerStarted","Data":"694105843a756a02f2ae1ec178b752708fd59a651c6c06e90e797492c1f46372"}
Oct 31 03:28:57.357695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:57.357287    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c" event={"ID":"dbd298c3-0d0b-48e3-8d28-f0033803ce09","Type":"ContainerStarted","Data":"dd3fd0f83245d65599b77b0ed699e3e7293d8ec2270f821051a2ea7601633558"}
Oct 31 03:28:57.471036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:57.470982    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c" podStartSLOduration=2.470966839 podStartE2EDuration="2.470966839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:57.466931356 +0000 UTC m=+1287.481493075" watchObservedRunningTime="2026-10-31 03:28:57.470966839 +0000 UTC m=+1287.485528571"
Oct 31 03:28:57.955907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:57.955869    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 03:28:57.956127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:57.955929    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 03:28:58.369212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:58.369112    5831 generic.go:356] "Generic (PLEG): container finished" podID="dbd298c3-0d0b-48e3-8d28-f0033803ce09" containerID="694105843a756a02f2ae1ec178b752708fd59a651c6c06e90e797492c1f46372" exitCode=0
Oct 31 03:28:58.369628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:58.369207    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c" event={"ID":"dbd298c3-0d0b-48e3-8d28-f0033803ce09","Type":"ContainerDied","Data":"694105843a756a02f2ae1ec178b752708fd59a651c6c06e90e797492c1f46372"}
Oct 31 03:28:58.960624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:58.960593    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:28:58.961726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:58.961702    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:28:59.555647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.555624    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:28:59.655967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.655877    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\") pod \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") "
Oct 31 03:28:59.655967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.655922    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\") pod \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") "
Oct 31 03:28:59.655967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.655950    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\" (UniqueName: \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\") pod \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\" (UID: \"dbd298c3-0d0b-48e3-8d28-f0033803ce09\") "
Oct 31 03:28:59.656378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.656266    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca" pod "dbd298c3-0d0b-48e3-8d28-f0033803ce09" (UID: "dbd298c3-0d0b-48e3-8d28-f0033803ce09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:59.656659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.656637    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:59.658326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.658301    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x" pod "dbd298c3-0d0b-48e3-8d28-f0033803ce09" (UID: "dbd298c3-0d0b-48e3-8d28-f0033803ce09"). InnerVolumeSpecName "kube-api-access-td26x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:59.658434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.658398    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca" pod "dbd298c3-0d0b-48e3-8d28-f0033803ce09" (UID: "dbd298c3-0d0b-48e3-8d28-f0033803ce09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:59.758189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.758155    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd298c3-0d0b-48e3-8d28-f0033803ce09-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:59.758189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:28:59.758184    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td26x\" (UniqueName: \"kubernetes.io/projected/dbd298c3-0d0b-48e3-8d28-f0033803ce09-kube-api-access-td26x\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:00.388695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:00.388665    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"
Oct 31 03:29:00.388695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:00.388683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c" event={"ID":"dbd298c3-0d0b-48e3-8d28-f0033803ce09","Type":"ContainerDied","Data":"dd3fd0f83245d65599b77b0ed699e3e7293d8ec2270f821051a2ea7601633558"}
Oct 31 03:29:00.388933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:00.388736    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3fd0f83245d65599b77b0ed699e3e7293d8ec2270f821051a2ea7601633558"
Oct 31 03:29:02.518990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.518956    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"]
Oct 31 03:29:02.519970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.519950    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dbd298c3-0d0b-48e3-8d28-f0033803ce09" containerName="revoke-keys"
Oct 31 03:29:02.520033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.519974    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd298c3-0d0b-48e3-8d28-f0033803ce09" containerName="revoke-keys"
Oct 31 03:29:02.520173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.520161    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd298c3-0d0b-48e3-8d28-f0033803ce09" containerName="revoke-keys"
Oct 31 03:29:02.524846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.524825    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.620554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.607387    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.624561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.621772    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmpw\" (UniqueName: \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.624561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.621950    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.624561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.614124    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"]
Oct 31 03:29:02.630054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.629826    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:29:02.630054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.629876    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:02.630054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.629955    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:02.630054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.630022    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:02.723811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.723770    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.724218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.724181    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.724414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.724393    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.724575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.724480    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmpw\" (UniqueName: \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.727215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.727187    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.778664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.778590    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmpw\" (UniqueName: \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\") pod \"maas-api-revoke-sub-1bbd0957-h4f6s\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:02.839542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:02.839505    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:03.216567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:03.216500    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"]
Oct 31 03:29:03.217838 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:29:03.217808    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c314b4c_cf85_4376_98aa_e19043b47dd3.slice/crio-8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b WatchSource:0}: Error finding container 8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b: Status 404 returned error can't find the container with id 8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b
Oct 31 03:29:03.421847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:03.421812    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s" event={"ID":"0c314b4c-cf85-4376-98aa-e19043b47dd3","Type":"ContainerStarted","Data":"8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b"}
Oct 31 03:29:03.921890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:03.921808    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:29:04.433453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:04.433418    5831 generic.go:356] "Generic (PLEG): container finished" podID="0c314b4c-cf85-4376-98aa-e19043b47dd3" containerID="212fb4311441a00a900262eb0c403b54f477bf3d9d5ce0b764aa18d008bacecf" exitCode=0
Oct 31 03:29:04.433729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:04.433513    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s" event={"ID":"0c314b4c-cf85-4376-98aa-e19043b47dd3","Type":"ContainerDied","Data":"212fb4311441a00a900262eb0c403b54f477bf3d9d5ce0b764aa18d008bacecf"}
Oct 31 03:29:06.154177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.153737    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:06.280556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.255096    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\") pod \"0c314b4c-cf85-4376-98aa-e19043b47dd3\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") "
Oct 31 03:29:06.281685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.281644    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\") pod \"0c314b4c-cf85-4376-98aa-e19043b47dd3\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") "
Oct 31 03:29:06.291789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.291758    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\" (UniqueName: \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\") pod \"0c314b4c-cf85-4376-98aa-e19043b47dd3\" (UID: \"0c314b4c-cf85-4376-98aa-e19043b47dd3\") "
Oct 31 03:29:06.314555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.297435    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca" pod "0c314b4c-cf85-4376-98aa-e19043b47dd3" (UID: "0c314b4c-cf85-4376-98aa-e19043b47dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:06.343964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.343901    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw" pod "0c314b4c-cf85-4376-98aa-e19043b47dd3" (UID: "0c314b4c-cf85-4376-98aa-e19043b47dd3"). InnerVolumeSpecName "kube-api-access-szmpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:06.421607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.414682    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca" pod "0c314b4c-cf85-4376-98aa-e19043b47dd3" (UID: "0c314b4c-cf85-4376-98aa-e19043b47dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:06.421607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.418356    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:06.421607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.418395    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c314b4c-cf85-4376-98aa-e19043b47dd3-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:06.421607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.418413    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmpw\" (UniqueName: \"kubernetes.io/projected/0c314b4c-cf85-4376-98aa-e19043b47dd3-kube-api-access-szmpw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:06.571554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.571179    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s" event={"ID":"0c314b4c-cf85-4376-98aa-e19043b47dd3","Type":"ContainerDied","Data":"8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b"}
Oct 31 03:29:06.571554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.571246    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8237393a663f33de194e5cc55d25dfd6f46ba70f702bd8defab5c8647b62107b"
Oct 31 03:29:06.571554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:06.571320    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"
Oct 31 03:29:09.077778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.077745    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:29:09.078425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.078410    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0c314b4c-cf85-4376-98aa-e19043b47dd3" containerName="revoke-keys"
Oct 31 03:29:09.078469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.078427    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c314b4c-cf85-4376-98aa-e19043b47dd3" containerName="revoke-keys"
Oct 31 03:29:09.078594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.078583    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c314b4c-cf85-4376-98aa-e19043b47dd3" containerName="revoke-keys"
Oct 31 03:29:09.083203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.083178    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.134744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.134711    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a16786-openshift-default-dockercfg-rp2zc\""
Oct 31 03:29:09.143398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.143355    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.143779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.143755    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.143944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.143928    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144062    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144181    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144342    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144476    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144603    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7js9w\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.144841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.144822    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.263967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.263767    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.264881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.264855    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.265718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.265699    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.266652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.266518    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.267571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.267551    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.268361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.267855    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.268618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.268592    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.269564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.269393    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.269990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.269970    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.270290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.270116    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.270606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.270513    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7js9w\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.272364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.268292    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.272723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.271509    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.275226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.275205    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.344653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.282583    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:29:09.386205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.386116    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.417738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.408883    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.444835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.444802    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7js9w\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.449162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.449075    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\") pod \"e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") " pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:09.701764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:09.701732    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:10.223454 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:29:10.223407    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39127796_4e80_49c5_ab27_950828b2c41a.slice/crio-aed3acc8c7ff8663869ae2171cc0a576c88cf81038abb0d9c58cd6a0a8816481 WatchSource:0}: Error finding container aed3acc8c7ff8663869ae2171cc0a576c88cf81038abb0d9c58cd6a0a8816481: Status 404 returned error can't find the container with id aed3acc8c7ff8663869ae2171cc0a576c88cf81038abb0d9c58cd6a0a8816481
Oct 31 03:29:10.225481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.225455    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:29:10.225898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.225868    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:10.225952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.225931    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:10.225989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.225959    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:10.645557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.645436    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerStarted","Data":"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"}
Oct 31 03:29:10.645557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.645506    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerStarted","Data":"aed3acc8c7ff8663869ae2171cc0a576c88cf81038abb0d9c58cd6a0a8816481"}
Oct 31 03:29:10.702314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.702285    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:10.758174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:10.758115    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podStartSLOduration=1.758101741 podStartE2EDuration="1.758101741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:10.755566763 +0000 UTC m=+1300.770128468" watchObservedRunningTime="2026-10-31 03:29:10.758101741 +0000 UTC m=+1300.772663449"
Oct 31 03:29:11.707746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:11.707720    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:12.661263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:12.661228    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:12.662150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:12.662132    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:29:12.979049 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:29:12.979007    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:29:17.411553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.408566    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:29:17.423621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.423590    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.447428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.445340    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:29:17.566556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.560030    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a16786\""
Oct 31 03:29:17.566556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.560754    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a16786-dockercfg-vmj9r\""
Oct 31 03:29:17.566556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.564399    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9pp\" (UniqueName: \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.566556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.564487    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.779642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.718485    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9pp\" (UniqueName: \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.779642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.718600    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.791563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.780626    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.791563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.788498    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:29:17.836520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.836456    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9pp\" (UniqueName: \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\") pod \"payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:17.846906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.846867    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:29:17.847060 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.846972    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:17.941490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.941414    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:17.941949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:17.941929    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxwt\" (UniqueName: \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.043639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.043519    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxwt\" (UniqueName: \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.043827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.043732    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.044425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.044398    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.057155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.057121    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:18.102908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.102877    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxwt\" (UniqueName: \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\") pod \"payload-processing-e2e-api-a-a16786-77487b7677-frnsn\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") " pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.169399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.169366    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.443262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.443234    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:29:18.443660 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:29:18.443334    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bebe86_9191_4735_9073_5eb2f03eb07d.slice/crio-59f327a9f19b10a168e38924ec0064bcbe4812040761f02dda8bd080870ec5c5 WatchSource:0}: Error finding container 59f327a9f19b10a168e38924ec0064bcbe4812040761f02dda8bd080870ec5c5: Status 404 returned error can't find the container with id 59f327a9f19b10a168e38924ec0064bcbe4812040761f02dda8bd080870ec5c5
Oct 31 03:29:18.569152 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:29:18.569124    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600a7d70_95aa_4747_b9ee_58f2f039cf7a.slice/crio-a43b557e522e570b697498f6c1fc1ea29291c9d6260778f68e78a740f4a91a78 WatchSource:0}: Error finding container a43b557e522e570b697498f6c1fc1ea29291c9d6260778f68e78a740f4a91a78: Status 404 returned error can't find the container with id a43b557e522e570b697498f6c1fc1ea29291c9d6260778f68e78a740f4a91a78
Oct 31 03:29:18.569830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.569807    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:29:18.888116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.886439    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" event={"ID":"600a7d70-95aa-4747-b9ee-58f2f039cf7a","Type":"ContainerStarted","Data":"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"}
Oct 31 03:29:18.888116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.886495    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" event={"ID":"600a7d70-95aa-4747-b9ee-58f2f039cf7a","Type":"ContainerStarted","Data":"a43b557e522e570b697498f6c1fc1ea29291c9d6260778f68e78a740f4a91a78"}
Oct 31 03:29:18.888116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.887038    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:18.889169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.889139    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" event={"ID":"35bebe86-9191-4735-9073-5eb2f03eb07d","Type":"ContainerStarted","Data":"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"}
Oct 31 03:29:18.889330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.889188    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" event={"ID":"35bebe86-9191-4735-9073-5eb2f03eb07d","Type":"ContainerStarted","Data":"59f327a9f19b10a168e38924ec0064bcbe4812040761f02dda8bd080870ec5c5"}
Oct 31 03:29:18.889330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.889277    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:18.992970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:18.992912    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" podStartSLOduration=1.992898158 podStartE2EDuration="1.992898158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:18.992697725 +0000 UTC m=+1309.007259445" watchObservedRunningTime="2026-10-31 03:29:18.992898158 +0000 UTC m=+1309.007459865"
Oct 31 03:29:19.101073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:19.101019    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" podStartSLOduration=2.101004458 podStartE2EDuration="2.101004458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:19.099159012 +0000 UTC m=+1309.113720725" watchObservedRunningTime="2026-10-31 03:29:19.101004458 +0000 UTC m=+1309.115566165"
Oct 31 03:29:29.901481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:29.901447    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:29:30.912829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:30.912802    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:29:33.049273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:33.049242    5831 scope.go:122] "RemoveContainer" containerID="8d8e8af8da2f1ce206e3b37f4f8fa76a78257e695eed1a164b0f3b54c59b710e"
Oct 31 03:29:33.060684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:33.060664    5831 scope.go:122] "RemoveContainer" containerID="ced7cfd9f72e44c6a4e8ee64fe877dbdc5584d33f84181cf457e07b5cd0b395f"
Oct 31 03:29:33.071641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:33.071615    5831 scope.go:122] "RemoveContainer" containerID="1e61df17cb62e52c6a5f9e284750fe5433f79c246624dcaf5c4eabab2b9a4155"
Oct 31 03:29:37.101710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.101669    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"]
Oct 31 03:29:37.106233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.106214    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.160514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.160481    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"]
Oct 31 03:29:37.199288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.199228    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:37.199288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.199233    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:37.199288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.199246    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:37.200393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.200366    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:29:37.213072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.213036    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbd2\" (UniqueName: \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.213327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.213292    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.213434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.213343    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.314647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.314612    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.314855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.314656    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.314855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.314765    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbd2\" (UniqueName: \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.315355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.315330    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.317189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.317171    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.369233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.369156    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbd2\" (UniqueName: \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\") pod \"maas-api-revoke-sub-ed63152d-djddc\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.420393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.420360    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:37.795994 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:29:37.795963    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aac3aa0_bfd6_4b75_b2fa_52a302bc6564.slice/crio-3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218 WatchSource:0}: Error finding container 3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218: Status 404 returned error can't find the container with id 3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218
Oct 31 03:29:37.796158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:37.796055    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"]
Oct 31 03:29:38.171899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:38.171854    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc" event={"ID":"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564","Type":"ContainerStarted","Data":"24b32962732c73b8131f32cf79b2d084f3fe6a3ddcf379bad31c905c5e4bb13a"}
Oct 31 03:29:38.172302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:38.171919    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc" event={"ID":"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564","Type":"ContainerStarted","Data":"3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218"}
Oct 31 03:29:38.285260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:38.285201    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc" podStartSLOduration=1.285187534 podStartE2EDuration="1.285187534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:38.279321366 +0000 UTC m=+1328.293883074" watchObservedRunningTime="2026-10-31 03:29:38.285187534 +0000 UTC m=+1328.299749242"
Oct 31 03:29:39.181506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:39.181467    5831 generic.go:356] "Generic (PLEG): container finished" podID="3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" containerID="24b32962732c73b8131f32cf79b2d084f3fe6a3ddcf379bad31c905c5e4bb13a" exitCode=0
Oct 31 03:29:39.181974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:39.181559    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc" event={"ID":"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564","Type":"ContainerDied","Data":"24b32962732c73b8131f32cf79b2d084f3fe6a3ddcf379bad31c905c5e4bb13a"}
Oct 31 03:29:40.369773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.369743    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:40.459884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.459793    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\") pod \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") "
Oct 31 03:29:40.460070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.459909    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\" (UniqueName: \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\") pod \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") "
Oct 31 03:29:40.460070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.460023    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\") pod \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\" (UID: \"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564\") "
Oct 31 03:29:40.460254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.460185    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca" pod "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" (UID: "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:40.460618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.460600    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:40.462226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.462195    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca" pod "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" (UID: "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:40.462324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.462249    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2" pod "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" (UID: "3aac3aa0-bfd6-4b75-b2fa-52a302bc6564"). InnerVolumeSpecName "kube-api-access-pkbd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:40.562268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.562236    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbd2\" (UniqueName: \"kubernetes.io/projected/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-kube-api-access-pkbd2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:40.562268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:40.562265    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:41.204518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:41.204483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"
Oct 31 03:29:41.204738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:41.204511    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc" event={"ID":"3aac3aa0-bfd6-4b75-b2fa-52a302bc6564","Type":"ContainerDied","Data":"3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218"}
Oct 31 03:29:41.204738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:41.204589    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c97b86d03a179a9b86303bf21cbe802c661ad2087ddda29783e6a4b570f7218"
Oct 31 03:29:42.036090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:42.036052    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:29:42.983295 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:29:42.983252    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:29:48.697461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.697410    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:29:48.699123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.699071    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" containerName="revoke-keys"
Oct 31 03:29:48.699284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.699268    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" containerName="revoke-keys"
Oct 31 03:29:48.699636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.699617    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" containerName="revoke-keys"
Oct 31 03:29:48.727553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.718272    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.804151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.804110    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4fdeb-openshift-default-dockercfg-rtwzj\""
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817272    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817339    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817409    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817507    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncqx\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817556    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817590    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817627    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817866    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.820558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.817897    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.888945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.888906    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:29:48.920154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.920106    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.920427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.920405    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.920617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.920599    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.920838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.920804    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncqx\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.920963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.920948    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.921076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.921064    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.921190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.921177    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.921346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.921333    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.921443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.921431    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.922133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.922109    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.927477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.923873    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.927477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.924512    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.927477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.925235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.954468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.935074    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:48.989555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:48.986283    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:49.121947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.068242    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncqx\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:49.144932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.144067    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:49.149275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.148620    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\") pod \"e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") " pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:49.382002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.381921    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:49.834979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.834922    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:29:49.861551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.860755    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:49.861551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.860859    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:49.861551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:49.860905    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:29:50.350306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:50.350274    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerStarted","Data":"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"}
Oct 31 03:29:50.350477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:50.350329    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerStarted","Data":"c966acf7352473ce3b9aaa479f969caa3589f69cc27a1d4782a7e8276e27467e"}
Oct 31 03:29:50.382975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:50.382897    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:50.469506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:50.469445    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podStartSLOduration=2.469423467 podStartE2EDuration="2.469423467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:50.461523408 +0000 UTC m=+1340.476085115" watchObservedRunningTime="2026-10-31 03:29:50.469423467 +0000 UTC m=+1340.483985176"
Oct 31 03:29:51.388690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:51.388661    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:52.370609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:52.370575    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:52.371546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:52.371514    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:29:59.309069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.308635    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:29:59.348561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.340914    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.432630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.432590    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:29:59.450265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.450221    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4fdeb\""
Oct 31 03:29:59.456159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.456135    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4fdeb-dockercfg-7gv65\""
Oct 31 03:29:59.459370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.459334    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.459713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.459693    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbwf\" (UniqueName: \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.516105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.516073    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:29:59.524847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.524819    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.529638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.529614    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:29:59.560810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.560746    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.560810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.560779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbwf\" (UniqueName: \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.560989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.560962    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.561125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.561099    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.561753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.561732    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.614194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.614157    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbwf\" (UniqueName: \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\") pod \"payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.662964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.662929    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.663136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.663072    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.663815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.663795    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.717026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.716993    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\") pod \"payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:29:59.746860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.746829    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:29:59.845042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:29:59.845011    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:30:00.149081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.149028    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:30:00.165134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.165101    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:00.171938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.171885    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.177091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.174503    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:00.191846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.191821    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:00.191988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.191948    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.197478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.197456    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:00.197654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.197628    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.204988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.204962    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.206110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.206077    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:00.211913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.211892    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.220495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.220467    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:30:00.220630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.220581    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:30:00.220630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.220605    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:30:00.238384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.238252    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:00.244990 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:00.244948    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b629f48_0b1d_470b_9869_08113dce69ec.slice/crio-af69d4e507520629632133f21e794e59124aa6edd57e56f339f695e16ad1921d WatchSource:0}: Error finding container af69d4e507520629632133f21e794e59124aa6edd57e56f339f695e16ad1921d: Status 404 returned error can't find the container with id af69d4e507520629632133f21e794e59124aa6edd57e56f339f695e16ad1921d
Oct 31 03:30:00.250732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.246821    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:00.255343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.255274    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:30:00.287868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.287523    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ll5\" (UniqueName: \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.287868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.287613    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.287868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.287665    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xpr\" (UniqueName: \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.287868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.287763    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zvn\" (UniqueName: \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.287868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.287808    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.288561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.288397    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.301642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.301569    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:00.307583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.307555    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:00.312522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.312496    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391197    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391348    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391393    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391441    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd6g\" (UniqueName: \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391517    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ll5\" (UniqueName: \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391569    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391600    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87k5\" (UniqueName: \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391644    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xpr\" (UniqueName: \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391680    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.391779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zvn\" (UniqueName: \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.392851    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.393594    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.394204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.393826    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.445548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.445498    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zvn\" (UniqueName: \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.445737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.445719    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xpr\" (UniqueName: \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.447915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.447893    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ll5\" (UniqueName: \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.493246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.493203    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.493508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.493277    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd6g\" (UniqueName: \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.493508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.493371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87k5\" (UniqueName: \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.493508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.493420    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.494037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.494013    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.494296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.494274    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.524893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.524856    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" event={"ID":"3b629f48-0b1d-470b-9869-08113dce69ec","Type":"ContainerStarted","Data":"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"}
Oct 31 03:30:00.525079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.524922    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" event={"ID":"3b629f48-0b1d-470b-9869-08113dce69ec","Type":"ContainerStarted","Data":"af69d4e507520629632133f21e794e59124aa6edd57e56f339f695e16ad1921d"}
Oct 31 03:30:00.525079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.524951    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:30:00.526767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.526738    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" event={"ID":"c6a254da-1023-420e-9556-63959b494ca0","Type":"ContainerStarted","Data":"5e0af7f452ca56229d71301257bcfc75e0c7abaa9a3cbf8c0bdeeebe4d2f9e4a"}
Oct 31 03:30:00.526901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.526789    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" event={"ID":"c6a254da-1023-420e-9556-63959b494ca0","Type":"ContainerStarted","Data":"57af2f2fccb4a725198d27a212f690866e6ff5851796183ef14635dc6fafab35"}
Oct 31 03:30:00.526901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.526815    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:30:00.532108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.532086    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:00.546976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.546942    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87k5\" (UniqueName: \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.547274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.547256    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd6g\" (UniqueName: \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\") pod \"maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.556629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.556608    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:00.594952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.593759    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:00.607129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.606811    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:00.806010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.805984    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:00.815912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.815851    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" podStartSLOduration=1.815832151 podStartE2EDuration="1.815832151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:00.806371346 +0000 UTC m=+1350.820933056" watchObservedRunningTime="2026-10-31 03:30:00.815832151 +0000 UTC m=+1350.830393860"
Oct 31 03:30:00.994868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.994806    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" podStartSLOduration=2.994783562 podStartE2EDuration="2.994783562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:00.953822908 +0000 UTC m=+1350.968384820" watchObservedRunningTime="2026-10-31 03:30:00.994783562 +0000 UTC m=+1351.009345272"
Oct 31 03:30:00.996670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:00.996637    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:01.027993 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:01.027955    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5cb69a_034f_4a98_860c_cb3792905d5f.slice/crio-8ec085ad2d7afc148b39d86b998ee75ef284261d701b083d14416acddbfe854f WatchSource:0}: Error finding container 8ec085ad2d7afc148b39d86b998ee75ef284261d701b083d14416acddbfe854f: Status 404 returned error can't find the container with id 8ec085ad2d7afc148b39d86b998ee75ef284261d701b083d14416acddbfe854f
Oct 31 03:30:01.045064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.045018    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:01.097378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.097346    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:01.135475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.135452    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:01.135724 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:01.135637    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36b74a1_c66d_49e3_8c1e_c8b3e0f943c9.slice/crio-a9532b2ddf534b06618b1875e044dba1a54a2dba53f8267eb74a72b1cab6205a WatchSource:0}: Error finding container a9532b2ddf534b06618b1875e044dba1a54a2dba53f8267eb74a72b1cab6205a: Status 404 returned error can't find the container with id a9532b2ddf534b06618b1875e044dba1a54a2dba53f8267eb74a72b1cab6205a
Oct 31 03:30:01.277134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.277036    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:01.536712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.536676    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerStarted","Data":"5e5c7e9be36f96583bead4dd67edc65fcc308a63e2917f460e420d5b6f275f8f"}
Oct 31 03:30:01.538165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.538140    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerStarted","Data":"a9532b2ddf534b06618b1875e044dba1a54a2dba53f8267eb74a72b1cab6205a"}
Oct 31 03:30:01.539503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.539476    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerStarted","Data":"8ec085ad2d7afc148b39d86b998ee75ef284261d701b083d14416acddbfe854f"}
Oct 31 03:30:01.540837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.540817    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerStarted","Data":"3d9f5d663df5da0466e02309b14828b675899ba3b09eec559fcd615bf3fc90c0"}
Oct 31 03:30:01.542125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:01.542103    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerStarted","Data":"babe52d168b5c527905ab632dd58e870aaeff4371f5038348001fb54d0d8deb2"}
Oct 31 03:30:02.552310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.552280    5831 generic.go:356] "Generic (PLEG): container finished" podID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerID="c52db9c00580f9834a98f9f9489fd0ba79e395d96da805ee28ac8ecac4f5bfae" exitCode=60
Oct 31 03:30:02.552867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.552366    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerDied","Data":"c52db9c00580f9834a98f9f9489fd0ba79e395d96da805ee28ac8ecac4f5bfae"}
Oct 31 03:30:02.552867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.552678    5831 scope.go:122] "RemoveContainer" containerID="c52db9c00580f9834a98f9f9489fd0ba79e395d96da805ee28ac8ecac4f5bfae"
Oct 31 03:30:02.554260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.554236    5831 generic.go:356] "Generic (PLEG): container finished" podID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerID="aa687add17a0450ae9e1dc76170bf0ee384bd88b18f78b20ba7936f1cc37c14f" exitCode=60
Oct 31 03:30:02.554454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.554311    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerDied","Data":"aa687add17a0450ae9e1dc76170bf0ee384bd88b18f78b20ba7936f1cc37c14f"}
Oct 31 03:30:02.554607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.554588    5831 scope.go:122] "RemoveContainer" containerID="aa687add17a0450ae9e1dc76170bf0ee384bd88b18f78b20ba7936f1cc37c14f"
Oct 31 03:30:02.557437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.557415    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerID="3b942576a1aa43a2734abdc54a01ff8aa995890802000b71426ffe51a4a4349a" exitCode=60
Oct 31 03:30:02.557613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.557509    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerDied","Data":"3b942576a1aa43a2734abdc54a01ff8aa995890802000b71426ffe51a4a4349a"}
Oct 31 03:30:02.557925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.557811    5831 scope.go:122] "RemoveContainer" containerID="3b942576a1aa43a2734abdc54a01ff8aa995890802000b71426ffe51a4a4349a"
Oct 31 03:30:02.559767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.559745    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerID="f275a2619eda0019557337c64f87bef169d75c0911c4da90d2c22c7908ed2331" exitCode=60
Oct 31 03:30:02.559878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.559814    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerDied","Data":"f275a2619eda0019557337c64f87bef169d75c0911c4da90d2c22c7908ed2331"}
Oct 31 03:30:02.559984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.559971    5831 scope.go:122] "RemoveContainer" containerID="f275a2619eda0019557337c64f87bef169d75c0911c4da90d2c22c7908ed2331"
Oct 31 03:30:02.561424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.561400    5831 generic.go:356] "Generic (PLEG): container finished" podID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerID="2f3666eefb1c13dfa2e8e8deb96c22dcb26ae21eaffe45ac6400ce1b65928337" exitCode=60
Oct 31 03:30:02.561511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.561456    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerDied","Data":"2f3666eefb1c13dfa2e8e8deb96c22dcb26ae21eaffe45ac6400ce1b65928337"}
Oct 31 03:30:02.561812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:02.561796    5831 scope.go:122] "RemoveContainer" containerID="2f3666eefb1c13dfa2e8e8deb96c22dcb26ae21eaffe45ac6400ce1b65928337"
Oct 31 03:30:03.575324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.575286    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerStarted","Data":"1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"}
Oct 31 03:30:03.578001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.577972    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerStarted","Data":"d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"}
Oct 31 03:30:03.586881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.586846    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerStarted","Data":"a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"}
Oct 31 03:30:03.590944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.590914    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerStarted","Data":"288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"}
Oct 31 03:30:03.594283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.594253    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerStarted","Data":"9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"}
Oct 31 03:30:03.714699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.711432    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" podStartSLOduration=3.711413017 podStartE2EDuration="3.711413017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:03.699966152 +0000 UTC m=+1353.714527860" watchObservedRunningTime="2026-10-31 03:30:03.711413017 +0000 UTC m=+1353.725974728"
Oct 31 03:30:03.810387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.810330    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" podStartSLOduration=3.810311628 podStartE2EDuration="3.810311628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:03.805047668 +0000 UTC m=+1353.819609378" watchObservedRunningTime="2026-10-31 03:30:03.810311628 +0000 UTC m=+1353.824873337"
Oct 31 03:30:03.914825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:03.914769    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" podStartSLOduration=3.914753819 podStartE2EDuration="3.914753819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:03.911013618 +0000 UTC m=+1353.925575327" watchObservedRunningTime="2026-10-31 03:30:03.914753819 +0000 UTC m=+1353.929315589"
Oct 31 03:30:04.021811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.021748    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" podStartSLOduration=4.02173009 podStartE2EDuration="4.02173009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:04.016000801 +0000 UTC m=+1354.030562506" watchObservedRunningTime="2026-10-31 03:30:04.02173009 +0000 UTC m=+1354.036291802"
Oct 31 03:30:04.124134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.124016    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" podStartSLOduration=4.123998842 podStartE2EDuration="4.123998842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:04.120189765 +0000 UTC m=+1354.134751472" watchObservedRunningTime="2026-10-31 03:30:04.123998842 +0000 UTC m=+1354.138560550"
Oct 31 03:30:04.608263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.608223    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerID="d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de" exitCode=60
Oct 31 03:30:04.608817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.608310    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerDied","Data":"d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"}
Oct 31 03:30:04.608817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.608382    5831 scope.go:122] "RemoveContainer" containerID="f275a2619eda0019557337c64f87bef169d75c0911c4da90d2c22c7908ed2331"
Oct 31 03:30:04.608817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.608737    5831 scope.go:122] "RemoveContainer" containerID="d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"
Oct 31 03:30:04.609212 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:04.609184    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh_odh-ai-gateway-infra(6e8366c4-3f8a-428f-aee6-53eae998f653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653"
Oct 31 03:30:04.610967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.610940    5831 generic.go:356] "Generic (PLEG): container finished" podID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerID="a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489" exitCode=60
Oct 31 03:30:04.611086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.611031    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerDied","Data":"a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"}
Oct 31 03:30:04.611301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.611287    5831 scope.go:122] "RemoveContainer" containerID="a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"
Oct 31 03:30:04.611677 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:04.611652    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll_odh-ai-gateway-infra(e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9"
Oct 31 03:30:04.613333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.613307    5831 generic.go:356] "Generic (PLEG): container finished" podID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerID="288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4" exitCode=60
Oct 31 03:30:04.613476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.613452    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerDied","Data":"288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"}
Oct 31 03:30:04.613706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.613684    5831 scope.go:122] "RemoveContainer" containerID="288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"
Oct 31 03:30:04.614047 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:04.614027    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5_odh-ai-gateway-infra(bf5cb69a-034f-4a98-860c-cb3792905d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f"
Oct 31 03:30:04.616018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.615996    5831 generic.go:356] "Generic (PLEG): container finished" podID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerID="9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc" exitCode=60
Oct 31 03:30:04.616121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.616075    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerDied","Data":"9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"}
Oct 31 03:30:04.616309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.616288    5831 scope.go:122] "RemoveContainer" containerID="9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"
Oct 31 03:30:04.616654 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:04.616616    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c_odh-ai-gateway-infra(51817f72-c255-4eeb-ae40-d50e78cf5936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936"
Oct 31 03:30:04.618422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.618400    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerID="1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07" exitCode=60
Oct 31 03:30:04.618561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.618476    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerDied","Data":"1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"}
Oct 31 03:30:04.618853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.618839    5831 scope.go:122] "RemoveContainer" containerID="1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"
Oct 31 03:30:04.619208 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:04.619186    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr_odh-ai-gateway-infra(0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6"
Oct 31 03:30:04.626224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.626205    5831 scope.go:122] "RemoveContainer" containerID="2f3666eefb1c13dfa2e8e8deb96c22dcb26ae21eaffe45ac6400ce1b65928337"
Oct 31 03:30:04.641254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.641235    5831 scope.go:122] "RemoveContainer" containerID="c52db9c00580f9834a98f9f9489fd0ba79e395d96da805ee28ac8ecac4f5bfae"
Oct 31 03:30:04.656571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.656512    5831 scope.go:122] "RemoveContainer" containerID="aa687add17a0450ae9e1dc76170bf0ee384bd88b18f78b20ba7936f1cc37c14f"
Oct 31 03:30:04.670697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:04.670681    5831 scope.go:122] "RemoveContainer" containerID="3b942576a1aa43a2734abdc54a01ff8aa995890802000b71426ffe51a4a4349a"
Oct 31 03:30:05.632131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:05.632104    5831 scope.go:122] "RemoveContainer" containerID="1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"
Oct 31 03:30:05.632644 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:05.632488    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr_odh-ai-gateway-infra(0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6"
Oct 31 03:30:05.633976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:05.633955    5831 scope.go:122] "RemoveContainer" containerID="d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"
Oct 31 03:30:05.634277 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:05.634257    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh_odh-ai-gateway-infra(6e8366c4-3f8a-428f-aee6-53eae998f653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653"
Oct 31 03:30:05.635514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:05.635492    5831 scope.go:122] "RemoveContainer" containerID="a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"
Oct 31 03:30:05.635786 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:05.635764    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll_odh-ai-gateway-infra(e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9"
Oct 31 03:30:05.637153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:05.637136    5831 scope.go:122] "RemoveContainer" containerID="288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"
Oct 31 03:30:05.637373 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:05.637352    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5_odh-ai-gateway-infra(bf5cb69a-034f-4a98-860c-cb3792905d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f"
Oct 31 03:30:05.638903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:05.638877    5831 scope.go:122] "RemoveContainer" containerID="9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"
Oct 31 03:30:05.639096 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:05.639080    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c_odh-ai-gateway-infra(51817f72-c255-4eeb-ae40-d50e78cf5936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936"
Oct 31 03:30:11.570764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:11.570730    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:30:11.618554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:11.608818    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:30:13.240555 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:13.231042    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:30:13.341552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:13.340694    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:14.137355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:14.137331    5831 scope.go:122] "RemoveContainer" containerID="d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"
Oct 31 03:30:14.870620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:14.870553    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerStarted","Data":"56849fa587ade33ccba64cdefc2b63822538fd30a72826275284e12f329b9529"}
Oct 31 03:30:15.127773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.127744    5831 scope.go:122] "RemoveContainer" containerID="9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"
Oct 31 03:30:15.883744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.883708    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerID="56849fa587ade33ccba64cdefc2b63822538fd30a72826275284e12f329b9529" exitCode=60
Oct 31 03:30:15.884183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.883781    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerDied","Data":"56849fa587ade33ccba64cdefc2b63822538fd30a72826275284e12f329b9529"}
Oct 31 03:30:15.884183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.883854    5831 scope.go:122] "RemoveContainer" containerID="d43297a290de3e39771aea07705afd1fe60d95a702a9db2c962e52881121c3de"
Oct 31 03:30:15.884745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.884727    5831 scope.go:122] "RemoveContainer" containerID="56849fa587ade33ccba64cdefc2b63822538fd30a72826275284e12f329b9529"
Oct 31 03:30:15.885320 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:15.885264    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh_odh-ai-gateway-infra(6e8366c4-3f8a-428f-aee6-53eae998f653)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653"
Oct 31 03:30:15.886323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:15.886297    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerStarted","Data":"353533d73b26ddc4439749af08eabed16461709eada6aa49f714404ae16b9fb7"}
Oct 31 03:30:16.070117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.070080    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:16.128666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.128635    5831 scope.go:122] "RemoveContainer" containerID="288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"
Oct 31 03:30:16.129338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.129317    5831 scope.go:122] "RemoveContainer" containerID="a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"
Oct 31 03:30:16.906230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.906144    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerStarted","Data":"5770be6944f564ac135ea89ecaeb562568dccf249feccfc58893e2472a575d75"}
Oct 31 03:30:16.909975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.909150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerStarted","Data":"7d044a569da39932726a29961de11cc9077fba7c9669f271d9c56241206d1c0b"}
Oct 31 03:30:16.911855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.911827    5831 generic.go:356] "Generic (PLEG): container finished" podID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerID="353533d73b26ddc4439749af08eabed16461709eada6aa49f714404ae16b9fb7" exitCode=60
Oct 31 03:30:16.912162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.911996    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerDied","Data":"353533d73b26ddc4439749af08eabed16461709eada6aa49f714404ae16b9fb7"}
Oct 31 03:30:16.912162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.912057    5831 scope.go:122] "RemoveContainer" containerID="9ace25d42a514226bd15f904f9391843f2bbc94ac552fe013f6bba33254040bc"
Oct 31 03:30:16.912523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:16.912505    5831 scope.go:122] "RemoveContainer" containerID="353533d73b26ddc4439749af08eabed16461709eada6aa49f714404ae16b9fb7"
Oct 31 03:30:16.913344 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:16.913319    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c_odh-ai-gateway-infra(51817f72-c255-4eeb-ae40-d50e78cf5936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936"
Oct 31 03:30:17.116148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.116118    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:17.127619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.127588    5831 scope.go:122] "RemoveContainer" containerID="1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"
Oct 31 03:30:17.168299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.168221    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:17.255453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.255420    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\" (UniqueName: \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\") pod \"6e8366c4-3f8a-428f-aee6-53eae998f653\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") "
Oct 31 03:30:17.255672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.255630    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\") pod \"6e8366c4-3f8a-428f-aee6-53eae998f653\" (UID: \"6e8366c4-3f8a-428f-aee6-53eae998f653\") "
Oct 31 03:30:17.256035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.256005    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca" pod "6e8366c4-3f8a-428f-aee6-53eae998f653" (UID: "6e8366c4-3f8a-428f-aee6-53eae998f653"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:17.256287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.256264    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8366c4-3f8a-428f-aee6-53eae998f653-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:17.280163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.280129    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn" pod "6e8366c4-3f8a-428f-aee6-53eae998f653" (UID: "6e8366c4-3f8a-428f-aee6-53eae998f653"). InnerVolumeSpecName "kube-api-access-w2zvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:17.357503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.357462    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zvn\" (UniqueName: \"kubernetes.io/projected/6e8366c4-3f8a-428f-aee6-53eae998f653-kube-api-access-w2zvn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:17.927955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.927870    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"
Oct 31 03:30:17.928442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.927861    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh" event={"ID":"6e8366c4-3f8a-428f-aee6-53eae998f653","Type":"ContainerDied","Data":"5e5c7e9be36f96583bead4dd67edc65fcc308a63e2917f460e420d5b6f275f8f"}
Oct 31 03:30:17.928442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.928023    5831 scope.go:122] "RemoveContainer" containerID="56849fa587ade33ccba64cdefc2b63822538fd30a72826275284e12f329b9529"
Oct 31 03:30:17.930662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.930634    5831 generic.go:356] "Generic (PLEG): container finished" podID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerID="5770be6944f564ac135ea89ecaeb562568dccf249feccfc58893e2472a575d75" exitCode=60
Oct 31 03:30:17.930817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.930707    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerDied","Data":"5770be6944f564ac135ea89ecaeb562568dccf249feccfc58893e2472a575d75"}
Oct 31 03:30:17.931163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.931143    5831 scope.go:122] "RemoveContainer" containerID="5770be6944f564ac135ea89ecaeb562568dccf249feccfc58893e2472a575d75"
Oct 31 03:30:17.931494 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:17.931469    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll_odh-ai-gateway-infra(e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9"
Oct 31 03:30:17.933302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.933021    5831 generic.go:356] "Generic (PLEG): container finished" podID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerID="7d044a569da39932726a29961de11cc9077fba7c9669f271d9c56241206d1c0b" exitCode=60
Oct 31 03:30:17.933302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.933057    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerDied","Data":"7d044a569da39932726a29961de11cc9077fba7c9669f271d9c56241206d1c0b"}
Oct 31 03:30:17.933590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.933570    5831 scope.go:122] "RemoveContainer" containerID="7d044a569da39932726a29961de11cc9077fba7c9669f271d9c56241206d1c0b"
Oct 31 03:30:17.933926 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:17.933901    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5_odh-ai-gateway-infra(bf5cb69a-034f-4a98-860c-cb3792905d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f"
Oct 31 03:30:17.938270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.938246    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerStarted","Data":"e2b08bac7f736dfba2e493f0a7d3d048f1bfb671f58f8f2a767f5594b00e41d9"}
Oct 31 03:30:17.942669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:17.942651    5831 scope.go:122] "RemoveContainer" containerID="a0661a0832324af356d4be6fa7699b40e8daea6d74d8ad29eee536775bed7489"
Oct 31 03:30:18.002928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.002868    5831 scope.go:122] "RemoveContainer" containerID="288e5c87d76c7dd04141b0a96742b5afcdcc680a02ac7c985943e26559a094e4"
Oct 31 03:30:18.147514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.147483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:18.150778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.150751    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:18.175875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.175837    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\" (UniqueName: \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\") pod \"51817f72-c255-4eeb-ae40-d50e78cf5936\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") "
Oct 31 03:30:18.176034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.175880    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\") pod \"51817f72-c255-4eeb-ae40-d50e78cf5936\" (UID: \"51817f72-c255-4eeb-ae40-d50e78cf5936\") "
Oct 31 03:30:18.176618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.176251    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca" pod "51817f72-c255-4eeb-ae40-d50e78cf5936" (UID: "51817f72-c255-4eeb-ae40-d50e78cf5936"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:18.176618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.176617    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51817f72-c255-4eeb-ae40-d50e78cf5936-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:18.178258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.178240    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g" pod "51817f72-c255-4eeb-ae40-d50e78cf5936" (UID: "51817f72-c255-4eeb-ae40-d50e78cf5936"). InnerVolumeSpecName "kube-api-access-rrd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:18.256340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.256307    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:18.277922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.277888    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrd6g\" (UniqueName: \"kubernetes.io/projected/51817f72-c255-4eeb-ae40-d50e78cf5936-kube-api-access-rrd6g\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:18.305303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.305267    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:18.313493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.313454    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890290-h8vgh"]
Oct 31 03:30:18.951096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.951061    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerID="e2b08bac7f736dfba2e493f0a7d3d048f1bfb671f58f8f2a767f5594b00e41d9" exitCode=60
Oct 31 03:30:18.951705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.951131    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerDied","Data":"e2b08bac7f736dfba2e493f0a7d3d048f1bfb671f58f8f2a767f5594b00e41d9"}
Oct 31 03:30:18.951705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.951221    5831 scope.go:122] "RemoveContainer" containerID="1f1ee2087954c78f95a873ebb0650273faa4dec362f69ca2c514c7bd23ac3b07"
Oct 31 03:30:18.951837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.951767    5831 scope.go:122] "RemoveContainer" containerID="e2b08bac7f736dfba2e493f0a7d3d048f1bfb671f58f8f2a767f5594b00e41d9"
Oct 31 03:30:18.952200 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:18.952158    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr_odh-ai-gateway-infra(0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6"
Oct 31 03:30:18.957893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.957870    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c" event={"ID":"51817f72-c255-4eeb-ae40-d50e78cf5936","Type":"ContainerDied","Data":"3d9f5d663df5da0466e02309b14828b675899ba3b09eec559fcd615bf3fc90c0"}
Oct 31 03:30:18.958001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.957979    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"
Oct 31 03:30:18.969950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:18.969931    5831 scope.go:122] "RemoveContainer" containerID="353533d73b26ddc4439749af08eabed16461709eada6aa49f714404ae16b9fb7"
Oct 31 03:30:19.109550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.104907    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:19.216644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.216498    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:19.217361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.217299    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:19.223048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.223010    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a16786-29890290-w6r9c"]
Oct 31 03:30:19.274238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.274212    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:19.315304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.315266    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\") pod \"bf5cb69a-034f-4a98-860c-cb3792905d5f\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") "
Oct 31 03:30:19.315559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.315356    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\" (UniqueName: \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\") pod \"bf5cb69a-034f-4a98-860c-cb3792905d5f\" (UID: \"bf5cb69a-034f-4a98-860c-cb3792905d5f\") "
Oct 31 03:30:19.315704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.315678    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca" pod "bf5cb69a-034f-4a98-860c-cb3792905d5f" (UID: "bf5cb69a-034f-4a98-860c-cb3792905d5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:19.315780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.315698    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\" (UniqueName: \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\") pod \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") "
Oct 31 03:30:19.315780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.315746    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\") pod \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\" (UID: \"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9\") "
Oct 31 03:30:19.316114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.316088    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca" pod "e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" (UID: "e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:19.316488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.316464    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:19.316642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.316491    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cb69a-034f-4a98-860c-cb3792905d5f-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:19.317906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.317883    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr" pod "bf5cb69a-034f-4a98-860c-cb3792905d5f" (UID: "bf5cb69a-034f-4a98-860c-cb3792905d5f"). InnerVolumeSpecName "kube-api-access-88xpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:19.317995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.317957    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5" pod "e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" (UID: "e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9"). InnerVolumeSpecName "kube-api-access-v6ll5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:19.417858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.417827    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xpr\" (UniqueName: \"kubernetes.io/projected/bf5cb69a-034f-4a98-860c-cb3792905d5f-kube-api-access-88xpr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:19.417858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.417856    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ll5\" (UniqueName: \"kubernetes.io/projected/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9-kube-api-access-v6ll5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:19.968870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.968828    5831 generic.go:356] "Generic (PLEG): container finished" podID="39127796-4e80-49c5-ab27-950828b2c41a" containerID="c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38" exitCode=0
Oct 31 03:30:19.969418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.968904    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerDied","Data":"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"}
Oct 31 03:30:19.969486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.969427    5831 scope.go:122] "RemoveContainer" containerID="c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"
Oct 31 03:30:19.970757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.970729    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"
Oct 31 03:30:19.970757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.970745    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll" event={"ID":"e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9","Type":"ContainerDied","Data":"a9532b2ddf534b06618b1875e044dba1a54a2dba53f8267eb74a72b1cab6205a"}
Oct 31 03:30:19.970966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.970813    5831 scope.go:122] "RemoveContainer" containerID="5770be6944f564ac135ea89ecaeb562568dccf249feccfc58893e2472a575d75"
Oct 31 03:30:19.971164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.971136    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:19.971234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.971209    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:19.971315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.971296    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:19.972522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.972501    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"
Oct 31 03:30:19.972522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.972543    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5" event={"ID":"bf5cb69a-034f-4a98-860c-cb3792905d5f","Type":"ContainerDied","Data":"8ec085ad2d7afc148b39d86b998ee75ef284261d701b083d14416acddbfe854f"}
Oct 31 03:30:19.995915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:19.995891    5831 scope.go:122] "RemoveContainer" containerID="7d044a569da39932726a29961de11cc9077fba7c9669f271d9c56241206d1c0b"
Oct 31 03:30:20.144309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.144280    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" path="/var/lib/kubelet/pods/51817f72-c255-4eeb-ae40-d50e78cf5936/volumes"
Oct 31 03:30:20.145024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.145002    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" path="/var/lib/kubelet/pods/6e8366c4-3f8a-428f-aee6-53eae998f653/volumes"
Oct 31 03:30:20.186398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.186360    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:20.239152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.239055    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\") pod \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") "
Oct 31 03:30:20.239335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.239234    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\" (UniqueName: \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\") pod \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\" (UID: \"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6\") "
Oct 31 03:30:20.242068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.242035    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca" pod "0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" (UID: "0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:20.248323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.248294    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5" pod "0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" (UID: "0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6"). InnerVolumeSpecName "kube-api-access-b87k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:20.315066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.315033    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:20.320693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.320664    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890290-vmwll"]
Oct 31 03:30:20.341485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.341450    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:20.341485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.341481    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87k5\" (UniqueName: \"kubernetes.io/projected/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6-kube-api-access-b87k5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:20.420339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.420308    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:20.425683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:20.425648    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890290-4m6l5"]
Oct 31 03:30:21.012398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.012365    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerStarted","Data":"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"}
Oct 31 03:30:21.016202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.016179    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"
Oct 31 03:30:21.016344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.016211    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr" event={"ID":"0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6","Type":"ContainerDied","Data":"babe52d168b5c527905ab632dd58e870aaeff4371f5038348001fb54d0d8deb2"}
Oct 31 03:30:21.016344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.016277    5831 scope.go:122] "RemoveContainer" containerID="e2b08bac7f736dfba2e493f0a7d3d048f1bfb671f58f8f2a767f5594b00e41d9"
Oct 31 03:30:21.047893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.047869    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"]
Oct 31 03:30:21.054044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.054013    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5b28f-4mvcl"]
Oct 31 03:30:21.102874 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:21.102828    5831 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa1a4c7_3f64_488d_9b7a_b8770ff14fa6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa1a4c7_3f64_488d_9b7a_b8770ff14fa6.slice/crio-babe52d168b5c527905ab632dd58e870aaeff4371f5038348001fb54d0d8deb2\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:21.226804 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.226761    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:21.232148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.232124    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890290-92kdr"]
Oct 31 03:30:21.709838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.709752    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:30:21.710043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.709995    5831 patch_prober.go:29] interesting pod/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 03:30:21.710117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:21.710057    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 03:30:22.136443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.136367    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" path="/var/lib/kubelet/pods/0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6/volumes"
Oct 31 03:30:22.136911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.136891    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38315051-94bb-49e4-a822-2e119454c0a6" path="/var/lib/kubelet/pods/38315051-94bb-49e4-a822-2e119454c0a6/volumes"
Oct 31 03:30:22.137279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.137260    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" path="/var/lib/kubelet/pods/bf5cb69a-034f-4a98-860c-cb3792905d5f/volumes"
Oct 31 03:30:22.138031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.138011    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" path="/var/lib/kubelet/pods/e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9/volumes"
Oct 31 03:30:22.702968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.702933    5831 patch_prober.go:29] interesting pod/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 03:30:22.703281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:22.703255    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 03:30:23.703999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:23.703957    5831 patch_prober.go:29] interesting pod/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 03:30:23.704626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:23.704590    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 03:30:24.704032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:24.703987    5831 patch_prober.go:29] interesting pod/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.704575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:24.704056    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 03:30:25.703966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:25.703924    5831 patch_prober.go:29] interesting pod/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 03:30:25.704157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:25.704020    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 03:30:26.706909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:26.706880    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:30:26.707420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:26.707404    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:30:26.708447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:26.708422    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:30:32.050863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:32.050819    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"]
Oct 31 03:30:32.094001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:32.093959    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5516e46-zlm84"]
Oct 31 03:30:32.136261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:32.136228    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3935c86a-5f18-4ade-83a0-ccddfe4e088d" path="/var/lib/kubelet/pods/3935c86a-5f18-4ade-83a0-ccddfe4e088d/volumes"
Oct 31 03:30:33.773086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:33.773047    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:30:41.507382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.507332    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:30:41.515850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.515814    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.516086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.516064    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.524338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524312    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.524568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524547    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.524695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524682    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.524791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524779    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.524907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524895    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.524998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.524985    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.525084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.525073    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.525165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.525154    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.531509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.525252    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.540916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.540887    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.541130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541112    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.541230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541218    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.541323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541314    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.541396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541386    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.541493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541484    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.541590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541578    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.541700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541690    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.541780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541771    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.541871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541861    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.541973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.541963    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.542306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542288    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.542418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542407    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.542635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542620    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.542729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542720    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.542807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542795    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.542929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.542909    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:41.544256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.544238    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:41.544395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.544382    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:41.544489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.544479    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.544594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.544582    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:41.544684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.544675    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:41.581164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.574833    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.631624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.626876    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-070d07-openshift-default-dockercfg-tcmc7\""
Oct 31 03:30:41.666408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.666362    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.666821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.666800    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.667070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.667049    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.667216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.667200    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.667441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.667427    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.667649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.667636    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.667859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.667786    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpksb\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.668029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.668011    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.668293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.668276    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.694669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.694607    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:30:41.770060 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.769971    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770068    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770114    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpksb\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770139    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770159    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770281    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770334    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770391    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770489    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.770887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770827    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.771028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770910    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.771076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.770971    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.772626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.772607    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.773107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.773090    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.877579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.875336    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpksb\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.877579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.875923    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\") pod \"e2e-api-b-070d07-openshift-default-65f7d87779-6qltx\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") " pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:41.965581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:41.965016    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:42.423235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:42.423149    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:30:42.458330 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:42.458297    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e453fae_fa07_4c20_ae2d_71b1abd1af98.slice/crio-47e3a745e4ae05ad6d4bf4a3a2b5a0413e25aa75cfc7ccfe3396e2e9ce03feae WatchSource:0}: Error finding container 47e3a745e4ae05ad6d4bf4a3a2b5a0413e25aa75cfc7ccfe3396e2e9ce03feae: Status 404 returned error can't find the container with id 47e3a745e4ae05ad6d4bf4a3a2b5a0413e25aa75cfc7ccfe3396e2e9ce03feae
Oct 31 03:30:42.461000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:42.460968    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:42.461075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:42.461035    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:42.461075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:42.461067    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:30:43.072233 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:30:43.071296    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:30:43.288719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:43.288675    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerStarted","Data":"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"}
Oct 31 03:30:43.288900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:43.288729    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerStarted","Data":"47e3a745e4ae05ad6d4bf4a3a2b5a0413e25aa75cfc7ccfe3396e2e9ce03feae"}
Oct 31 03:30:43.401594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:43.401442    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podStartSLOduration=2.401422913 podStartE2EDuration="2.401422913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:43.400808437 +0000 UTC m=+1393.415370145" watchObservedRunningTime="2026-10-31 03:30:43.401422913 +0000 UTC m=+1393.415984629"
Oct 31 03:30:43.965907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:43.965869    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:43.971850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:43.971820    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:44.303278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:44.303241    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:44.304365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:44.304343    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:30:48.053326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:48.053240    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"]
Oct 31 03:30:48.095004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:48.094965    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41595c5b-cm6sd"]
Oct 31 03:30:48.136986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:48.136951    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdde98f2-edd5-4ead-8716-92a61b3b6953" path="/var/lib/kubelet/pods/cdde98f2-edd5-4ead-8716-92a61b3b6953/volumes"
Oct 31 03:30:49.967651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.967617    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:30:49.968302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968288    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:49.968350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968304    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa1a4c7-3f64-488d-9b7a-b8770ff14fa6" containerName="cleanup"
Oct 31 03:30:49.968350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968333    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:49.968350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968338    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:49.968454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968361    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:49.968454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968366    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:49.968454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968379    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:49.968454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968384    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:49.968630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968485    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36b74a1-c66d-49e3-8c1e-c8b3e0f943c9" containerName="cleanup"
Oct 31 03:30:49.968630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968496    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51817f72-c255-4eeb-ae40-d50e78cf5936" containerName="cleanup"
Oct 31 03:30:49.968630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968503    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5cb69a-034f-4a98-860c-cb3792905d5f" containerName="cleanup"
Oct 31 03:30:49.968630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.968512    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8366c4-3f8a-428f-aee6-53eae998f653" containerName="cleanup"
Oct 31 03:30:49.973716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:49.973696    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.026108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.026046    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:30:50.071557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.067830    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-070d07\""
Oct 31 03:30:50.071557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.067982    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-070d07-dockercfg-fxppc\""
Oct 31 03:30:50.153163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.134512    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.153163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.134780    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcgl\" (UniqueName: \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.295607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.289650    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.295607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.289951    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcgl\" (UniqueName: \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.295607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.291299    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.432451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.432383    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcgl\" (UniqueName: \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\") pod \"payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.462542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.462487    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:30:50.469001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.468958    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:30:50.469193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.469068    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.540764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.540726    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhrt\" (UniqueName: \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.540964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.540876    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.589881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.589793    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:50.642192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.642158    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrt\" (UniqueName: \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.642360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.642279    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.642975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.642955    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.696326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.696264    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhrt\" (UniqueName: \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\") pod \"payload-processing-e2e-api-b-070d07-6487b76589-zkd5m\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") " pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.788316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.788278    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:50.972799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:50.972762    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:30:50.973299 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:50.973273    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7822da_6223_4e38_9216_cef279a067c9.slice/crio-762607cf6dbb63015c40d711a426b694b48dd95a0481a62881ecc8f9224e2c3b WatchSource:0}: Error finding container 762607cf6dbb63015c40d711a426b694b48dd95a0481a62881ecc8f9224e2c3b: Status 404 returned error can't find the container with id 762607cf6dbb63015c40d711a426b694b48dd95a0481a62881ecc8f9224e2c3b
Oct 31 03:30:51.046858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.046833    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"]
Oct 31 03:30:51.051909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.051882    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff8be89-mxtrh"]
Oct 31 03:30:51.168180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.168152    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:30:51.169883 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:30:51.169849    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65f3b0d_d812_4b9a_9c42_78d8cf54e95a.slice/crio-a8536dd791d6ef2943e996677d8375a164de73f2a87a4de41913c2c805c7b790 WatchSource:0}: Error finding container a8536dd791d6ef2943e996677d8375a164de73f2a87a4de41913c2c805c7b790: Status 404 returned error can't find the container with id a8536dd791d6ef2943e996677d8375a164de73f2a87a4de41913c2c805c7b790
Oct 31 03:30:51.470959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.470915    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" event={"ID":"4b7822da-6223-4e38-9216-cef279a067c9","Type":"ContainerStarted","Data":"8f2bf77b6c3c8f7987759cc448e7718c80b56df5504e548bb7a5e30255a24357"}
Oct 31 03:30:51.471157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.471014    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" event={"ID":"4b7822da-6223-4e38-9216-cef279a067c9","Type":"ContainerStarted","Data":"762607cf6dbb63015c40d711a426b694b48dd95a0481a62881ecc8f9224e2c3b"}
Oct 31 03:30:51.471222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.471167    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:30:51.472756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.472727    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" event={"ID":"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a","Type":"ContainerStarted","Data":"9b851b4675a4b1ebf2ed5bc182803365a539a16c4c6b5cd84eace3f97a95818d"}
Oct 31 03:30:51.472854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.472769    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" event={"ID":"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a","Type":"ContainerStarted","Data":"a8536dd791d6ef2943e996677d8375a164de73f2a87a4de41913c2c805c7b790"}
Oct 31 03:30:51.472854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.472821    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:30:51.576238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.576182    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" podStartSLOduration=2.576167839 podStartE2EDuration="2.576167839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:51.574997159 +0000 UTC m=+1401.589558867" watchObservedRunningTime="2026-10-31 03:30:51.576167839 +0000 UTC m=+1401.590729561"
Oct 31 03:30:51.684409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:51.684343    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" podStartSLOduration=1.6843236959999999 podStartE2EDuration="1.684323696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:51.680903961 +0000 UTC m=+1401.695465670" watchObservedRunningTime="2026-10-31 03:30:51.684323696 +0000 UTC m=+1401.698885405"
Oct 31 03:30:52.138291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:30:52.138258    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bc629d-37c0-4aa6-9bf5-b6554f898b50" path="/var/lib/kubelet/pods/82bc629d-37c0-4aa6-9bf5-b6554f898b50/volumes"
Oct 31 03:31:02.485981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:02.485953    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:31:02.486395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:02.486025    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:31:03.057790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:03.057748    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"]
Oct 31 03:31:03.062340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:03.062314    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516fa178-548kn"]
Oct 31 03:31:04.138063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.138022    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14709779-5f9a-46f0-8083-453b0a892736" path="/var/lib/kubelet/pods/14709779-5f9a-46f0-8083-453b0a892736/volumes"
Oct 31 03:31:04.684788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.684752    5831 generic.go:356] "Generic (PLEG): container finished" podID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerID="9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864" exitCode=0
Oct 31 03:31:04.685022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.684827    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerDied","Data":"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"}
Oct 31 03:31:04.685296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.685279    5831 scope.go:122] "RemoveContainer" containerID="9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"
Oct 31 03:31:04.686886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.686858    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:04.686963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.686931    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:04.687001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:04.686963    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:05.711437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:05.711406    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerStarted","Data":"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"}
Oct 31 03:31:06.046332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.046235    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"]
Oct 31 03:31:06.051119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.051090    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580517da-lckr6"]
Oct 31 03:31:06.146837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.146797    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c8bfe6-beab-4493-9f4f-3e98944d2b5b" path="/var/lib/kubelet/pods/41c8bfe6-beab-4493-9f4f-3e98944d2b5b/volumes"
Oct 31 03:31:06.389850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.382854    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:31:06.404580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.403801    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 03:31:06.404580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:06.403900    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 03:31:07.371639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.371609    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:31:07.395584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.383063    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 03:31:07.395584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.383141    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 03:31:07.657603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.657495    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"]
Oct 31 03:31:07.662716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.662693    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.717310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.717275    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"]
Oct 31 03:31:07.756657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.756618    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:07.757017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.756968    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:07.757017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.756996    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:07.757017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.757005    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:31:07.776661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.776621    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.776855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.776832    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.776951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.776933    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vc5m\" (UniqueName: \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.878579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.878484    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.878791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.878634    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.878791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.878704    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vc5m\" (UniqueName: \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.904558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.879917    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.908436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.908368    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.931926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.931898    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vc5m\" (UniqueName: \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\") pod \"maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:07.977387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:07.977352    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:08.397555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:08.382934    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 03:31:08.397555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:08.382993    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 03:31:08.412126 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:08.412083    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fd5ca8_e505_46fe_a582_acdc78d2e8a8.slice/crio-b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49 WatchSource:0}: Error finding container b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49: Status 404 returned error can't find the container with id b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49
Oct 31 03:31:08.413512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:08.413479    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"]
Oct 31 03:31:08.828589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:08.824357    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6" event={"ID":"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8","Type":"ContainerStarted","Data":"b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49"}
Oct 31 03:31:09.397554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:09.386979    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 03:31:09.397554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:09.387046    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 03:31:09.834816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:09.834779    5831 generic.go:356] "Generic (PLEG): container finished" podID="a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" containerID="e3e0a4f1499c9dd735def5f29872c951af0cd25698b4e220ee27b40f3b359b9e" exitCode=0
Oct 31 03:31:09.835034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:09.834859    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6" event={"ID":"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8","Type":"ContainerDied","Data":"e3e0a4f1499c9dd735def5f29872c951af0cd25698b4e220ee27b40f3b359b9e"}
Oct 31 03:31:10.386554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:10.386054    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 03:31:10.386554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:10.386118    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 03:31:11.030363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.030335    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:11.165266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.165183    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\" (UniqueName: \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\") pod \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") "
Oct 31 03:31:11.165266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.165248    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\") pod \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") "
Oct 31 03:31:11.165578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.165361    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\") pod \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\" (UID: \"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8\") "
Oct 31 03:31:11.165864 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.165821    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca" pod "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" (UID: "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:11.166132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.166111    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.168342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.168246    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca" pod "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" (UID: "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:11.168458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.168393    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m" pod "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" (UID: "a5fd5ca8-e505-46fe-a582-acdc78d2e8a8"). InnerVolumeSpecName "kube-api-access-4vc5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:11.267633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.267590    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vc5m\" (UniqueName: \"kubernetes.io/projected/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-kube-api-access-4vc5m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.267633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.267625    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.387390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.387358    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:31:11.388791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.388769    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:31:11.854990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.854948    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6" event={"ID":"a5fd5ca8-e505-46fe-a582-acdc78d2e8a8","Type":"ContainerDied","Data":"b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49"}
Oct 31 03:31:11.854990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.854981    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"
Oct 31 03:31:11.855203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:11.855014    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b430f60352d83ff544a2ea8764490241bc89364a04513a3850417e000c406f49"
Oct 31 03:31:12.980670 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:31:12.980635    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:31:18.357555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:18.355477    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:31:19.045998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.045963    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"]
Oct 31 03:31:19.050685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.050653    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71525716-jjkk5"]
Oct 31 03:31:19.620551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.620288    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:31:19.852553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.840329    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"]
Oct 31 03:31:19.852553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.841575    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" containerName="revoke-keys"
Oct 31 03:31:19.852553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.841606    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" containerName="revoke-keys"
Oct 31 03:31:19.852553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.841939    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" containerName="revoke-keys"
Oct 31 03:31:19.881600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.876856    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:19.941611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.931254    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"]
Oct 31 03:31:19.941611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.937727    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:19.941611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.937887    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:31:19.941611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.938461    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:19.941611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:19.938524    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:20.025240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.023564    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.025240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.023621    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fpqw\" (UniqueName: \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.025240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.023853    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.241938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.241880    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.242306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.242265    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.242488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.242470    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fpqw\" (UniqueName: \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.263562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.263449    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.332649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.312653    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fpqw\" (UniqueName: \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.334104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.333282    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b7446d-52mtr\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:20.484960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.484922    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f414b21-bbc5-4b39-af24-6520f6a1f325" path="/var/lib/kubelet/pods/5f414b21-bbc5-4b39-af24-6520f6a1f325/volumes"
Oct 31 03:31:20.614862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:20.614768    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:21.183191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:21.183163    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"]
Oct 31 03:31:21.183599 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:21.183434    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccabe6f4_34b4_4861_9972_31270f6777da.slice/crio-0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56 WatchSource:0}: Error finding container 0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56: Status 404 returned error can't find the container with id 0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56
Oct 31 03:31:21.238340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:21.238306    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr" event={"ID":"ccabe6f4-34b4-4861-9972-31270f6777da","Type":"ContainerStarted","Data":"0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56"}
Oct 31 03:31:22.045989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:22.045954    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"]
Oct 31 03:31:22.091602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:22.091557    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10762baa-jnzjh"]
Oct 31 03:31:22.138615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:22.138575    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38" path="/var/lib/kubelet/pods/3d7a1fe8-f84f-44a1-aaca-a2df9dd87d38/volumes"
Oct 31 03:31:22.253175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:22.253137    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr" event={"ID":"ccabe6f4-34b4-4861-9972-31270f6777da","Type":"ContainerStarted","Data":"7bba6a535cd773a04fb8f7bb74fd161e295ae667e98806af52dfe1b3eb91bac5"}
Oct 31 03:31:23.267518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.267476    5831 generic.go:356] "Generic (PLEG): container finished" podID="ccabe6f4-34b4-4861-9972-31270f6777da" containerID="7bba6a535cd773a04fb8f7bb74fd161e295ae667e98806af52dfe1b3eb91bac5" exitCode=0
Oct 31 03:31:23.268030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.267605    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr" event={"ID":"ccabe6f4-34b4-4861-9972-31270f6777da","Type":"ContainerDied","Data":"7bba6a535cd773a04fb8f7bb74fd161e295ae667e98806af52dfe1b3eb91bac5"}
Oct 31 03:31:23.357648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.357594    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing" containerID="cri-o://17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5" gracePeriod=25
Oct 31 03:31:23.449721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.449693    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:23.470081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.470044    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\") pod \"ccabe6f4-34b4-4861-9972-31270f6777da\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") "
Oct 31 03:31:23.470264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.470119    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\") pod \"ccabe6f4-34b4-4861-9972-31270f6777da\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") "
Oct 31 03:31:23.470264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.470185    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\" (UniqueName: \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\") pod \"ccabe6f4-34b4-4861-9972-31270f6777da\" (UID: \"ccabe6f4-34b4-4861-9972-31270f6777da\") "
Oct 31 03:31:23.471556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.470518    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca" pod "ccabe6f4-34b4-4861-9972-31270f6777da" (UID: "ccabe6f4-34b4-4861-9972-31270f6777da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:23.471556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.470952    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:23.472820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.472795    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca" pod "ccabe6f4-34b4-4861-9972-31270f6777da" (UID: "ccabe6f4-34b4-4861-9972-31270f6777da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:23.473117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.473096    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw" pod "ccabe6f4-34b4-4861-9972-31270f6777da" (UID: "ccabe6f4-34b4-4861-9972-31270f6777da"). InnerVolumeSpecName "kube-api-access-9fpqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:23.572579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.572478    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccabe6f4-34b4-4861-9972-31270f6777da-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:23.572579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:23.572507    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fpqw\" (UniqueName: \"kubernetes.io/projected/ccabe6f4-34b4-4861-9972-31270f6777da-kube-api-access-9fpqw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:24.320295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:24.320264    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"
Oct 31 03:31:24.320837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:24.320275    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr" event={"ID":"ccabe6f4-34b4-4861-9972-31270f6777da","Type":"ContainerDied","Data":"0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56"}
Oct 31 03:31:24.320837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:24.320421    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6c8c54ec75c485aca08451f4d8a478abbb043d6999fbab4dd79be7782d3c56"
Oct 31 03:31:28.679149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:28.679041    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:31:31.400989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.400955    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:31:31.401808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.401791    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ccabe6f4-34b4-4861-9972-31270f6777da" containerName="revoke-keys"
Oct 31 03:31:31.401877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.401810    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccabe6f4-34b4-4861-9972-31270f6777da" containerName="revoke-keys"
Oct 31 03:31:31.401925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.401916    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccabe6f4-34b4-4861-9972-31270f6777da" containerName="revoke-keys"
Oct 31 03:31:31.407084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.407066    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.456194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.456162    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2260d8-openshift-default-dockercfg-qpfmm\""
Oct 31 03:31:31.508099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.508062    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:31:31.524214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.524168    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.524657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.524630    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.524928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.524908    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.525184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.525168    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.525352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.525338    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.525611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.525591    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkpl\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.525814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.525797    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.525966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.525950    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.526155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.526138    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686181    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686247    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686334    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkpl\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686390    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686424    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686455    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686571    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686664    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.686732    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.724613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.687237    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.743566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.735191    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.786503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.786468    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.800558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.793600    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.811364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.811097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.811364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.811321    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.811882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.811859    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.843267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.839027    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkpl\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:31.843267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:31.843191    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\") pod \"e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") " pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:32.053703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:32.033824    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:32.499139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:32.499103    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:31:32.531724 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:32.531694    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae2bd00_09a0_4542_84a7_980bb916168e.slice/crio-bb39164dec7373f05bcac805577bd11b6ee7c3bee6fdbed0588477c3b66375ee WatchSource:0}: Error finding container bb39164dec7373f05bcac805577bd11b6ee7c3bee6fdbed0588477c3b66375ee: Status 404 returned error can't find the container with id bb39164dec7373f05bcac805577bd11b6ee7c3bee6fdbed0588477c3b66375ee
Oct 31 03:31:32.534297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:32.534264    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:32.534401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:32.534334    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:32.534401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:32.534363    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:31:33.255728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:33.255690    5831 scope.go:122] "RemoveContainer" containerID="eb62b933fc2e65f4bc6cda8c82ba0b2f8b3539c36e74d650ab8f4f7587daa58a"
Oct 31 03:31:33.317046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:33.317018    5831 scope.go:122] "RemoveContainer" containerID="2b27cf4da9e1874029502383bfe10060b0bbc320905e22b7964297a4ad80a820"
Oct 31 03:31:33.407553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:33.406181    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerStarted","Data":"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"}
Oct 31 03:31:33.407553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:33.406301    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerStarted","Data":"bb39164dec7373f05bcac805577bd11b6ee7c3bee6fdbed0588477c3b66375ee"}
Oct 31 03:31:33.608600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:33.608427    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podStartSLOduration=2.608407992 podStartE2EDuration="2.608407992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:33.607977997 +0000 UTC m=+1443.622539705" watchObservedRunningTime="2026-10-31 03:31:33.608407992 +0000 UTC m=+1443.622969701"
Oct 31 03:31:34.034930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.034890    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:34.041619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.041583    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:34.047263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.047232    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"]
Oct 31 03:31:34.052435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.052410    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbb973a-p2l44"]
Oct 31 03:31:34.137141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.137110    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23" path="/var/lib/kubelet/pods/5d28a9b9-3f32-4f7c-b603-0d2ec7bd1d23/volumes"
Oct 31 03:31:34.433074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.433045    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:34.434439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:34.434410    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:31:38.664131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:38.664093    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:31:39.048016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:39.047972    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"]
Oct 31 03:31:39.095671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:39.095631    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846fcd47-m8pnq"]
Oct 31 03:31:40.138432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:40.138394    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7800029-029a-415f-8938-62dbde9fb104" path="/var/lib/kubelet/pods/c7800029-029a-415f-8938-62dbde9fb104/volumes"
Oct 31 03:31:42.337228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.330016    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:31:42.351676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.351633    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.423290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.423162    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4kv\" (UniqueName: \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.423290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.423278    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.436965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.432811    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2260d8\""
Oct 31 03:31:42.436965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.432884    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2260d8-dockercfg-wltts\""
Oct 31 03:31:42.444324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.444283    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:31:42.554134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.554083    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4kv\" (UniqueName: \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.554357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.554217    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.555270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.555239    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.576734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.571616    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:31:42.807568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.807334    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4kv\" (UniqueName: \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\") pod \"payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.893636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.893604    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:42.906091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.906064    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:42.997786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:42.972246    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:31:43.037050 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:31:43.037010    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:31:43.138287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.138245    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.138494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.138407    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lcs\" (UniqueName: \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.239991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.239957    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.240175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.240152    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lcs\" (UniqueName: \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.240742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.240722    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.294578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.294551    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lcs\" (UniqueName: \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\") pod \"payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.304467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.304440    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:43.392990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.392955    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:31:43.423740 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:43.423697    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd836abeb_2c33_4b93_90f9_db595d9003f7.slice/crio-f9f411302a7d1e55e0e1fb1665707e04d26baeefdda4cfd560ffbe62fdef509a WatchSource:0}: Error finding container f9f411302a7d1e55e0e1fb1665707e04d26baeefdda4cfd560ffbe62fdef509a: Status 404 returned error can't find the container with id f9f411302a7d1e55e0e1fb1665707e04d26baeefdda4cfd560ffbe62fdef509a
Oct 31 03:31:43.650851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.650821    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:31:43.705260 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:43.705230    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f44818_267b_4a30_9ed6_dd850f7e58ae.slice/crio-01f6f664d9b42bf264dbdcd82f0ccb49598cbb4ad704be6060d177db8e727a81 WatchSource:0}: Error finding container 01f6f664d9b42bf264dbdcd82f0ccb49598cbb4ad704be6060d177db8e727a81: Status 404 returned error can't find the container with id 01f6f664d9b42bf264dbdcd82f0ccb49598cbb4ad704be6060d177db8e727a81
Oct 31 03:31:43.705611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.705589    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:31:43.748464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.748429    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\") pod \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") "
Oct 31 03:31:43.748618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.748480    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\" (UniqueName: \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\") pod \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\" (UID: \"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30\") "
Oct 31 03:31:43.748871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.748841    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume" pod "2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" (UID: "2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:43.749294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.749247    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:43.750814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.750789    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2" pod "2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" (UID: "2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30"). InnerVolumeSpecName "kube-api-access-dh6d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:43.850337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:43.850304    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6d2\" (UniqueName: \"kubernetes.io/projected/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30-kube-api-access-dh6d2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:44.010634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.010600    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" event={"ID":"d836abeb-2c33-4b93-90f9-db595d9003f7","Type":"ContainerStarted","Data":"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"}
Oct 31 03:31:44.010829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.010665    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" event={"ID":"d836abeb-2c33-4b93-90f9-db595d9003f7","Type":"ContainerStarted","Data":"f9f411302a7d1e55e0e1fb1665707e04d26baeefdda4cfd560ffbe62fdef509a"}
Oct 31 03:31:44.010829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.010698    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:44.012404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.012374    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" event={"ID":"23f44818-267b-4a30-9ed6-dd850f7e58ae","Type":"ContainerStarted","Data":"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"}
Oct 31 03:31:44.012565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.012426    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" event={"ID":"23f44818-267b-4a30-9ed6-dd850f7e58ae","Type":"ContainerStarted","Data":"01f6f664d9b42bf264dbdcd82f0ccb49598cbb4ad704be6060d177db8e727a81"}
Oct 31 03:31:44.012565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.012456    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:44.013993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.013970    5831 generic.go:356] "Generic (PLEG): container finished" podID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerID="17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5" exitCode=0
Oct 31 03:31:44.014124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.014040    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" event={"ID":"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30","Type":"ContainerDied","Data":"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"}
Oct 31 03:31:44.014124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.014078    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"
Oct 31 03:31:44.014124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.014093    5831 scope.go:122] "RemoveContainer" containerID="17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"
Oct 31 03:31:44.014296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.014079    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf" event={"ID":"2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30","Type":"ContainerDied","Data":"b38bd4a64c17e9a1b0fa6ad5521eeb62a1139a01022a71a068f2fe46715e528a"}
Oct 31 03:31:44.026374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.026355    5831 scope.go:122] "RemoveContainer" containerID="17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"
Oct 31 03:31:44.026677 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:31:44.026653    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5\": container with ID starting with 17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5 not found: ID does not exist" containerID="17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"
Oct 31 03:31:44.026746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.026690    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5"} err="failed to get container status \"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5\": rpc error: code = NotFound desc = could not find container \"17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5\": container with ID starting with 17ff41806ea2f278e122f1fd4f466f3dc5a736076cfea8585d1e9787cbcc56f5 not found: ID does not exist"
Oct 31 03:31:44.121003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.120939    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" podStartSLOduration=2.120920451 podStartE2EDuration="2.120920451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:44.116710859 +0000 UTC m=+1454.131272567" watchObservedRunningTime="2026-10-31 03:31:44.120920451 +0000 UTC m=+1454.135482160"
Oct 31 03:31:44.227516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.227450    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" podStartSLOduration=2.227428221 podStartE2EDuration="2.227428221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:44.221900175 +0000 UTC m=+1454.236461882" watchObservedRunningTime="2026-10-31 03:31:44.227428221 +0000 UTC m=+1454.241989940"
Oct 31 03:31:44.380606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.376273    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:31:44.402552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:44.402030    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-5fhrf"]
Oct 31 03:31:45.079635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.079590    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:31:45.089985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.089953    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing"
Oct 31 03:31:45.090258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.090241    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing"
Oct 31 03:31:45.090717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.090683    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" containerName="payload-pre-processing"
Oct 31 03:31:45.109381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.109306    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.164124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.158588    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5effde-dockercfg-bzd2k\""
Oct 31 03:31:45.164124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.158677    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5effde\""
Oct 31 03:31:45.312074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.311684    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:31:45.322306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.322021    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.322306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.322281    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkg4h\" (UniqueName: \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.428665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.424630    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.428665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.424836    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkg4h\" (UniqueName: \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.457951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.443841    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.556550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.555820    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkg4h\" (UniqueName: \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\") pod \"payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:45.628709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:45.628672    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:46.007786 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:46.007753    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b56c5a7_b345_46a9_bf9c_27a67c25a64b.slice/crio-aac93b64eb2f962fd902d891e56b2f5cbe606c8998d86f8eb535fc4bf9ecabf9 WatchSource:0}: Error finding container aac93b64eb2f962fd902d891e56b2f5cbe606c8998d86f8eb535fc4bf9ecabf9: Status 404 returned error can't find the container with id aac93b64eb2f962fd902d891e56b2f5cbe606c8998d86f8eb535fc4bf9ecabf9
Oct 31 03:31:46.009854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:46.009826    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:31:46.087194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:46.087150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" event={"ID":"3b56c5a7-b345-46a9-bf9c-27a67c25a64b","Type":"ContainerStarted","Data":"aac93b64eb2f962fd902d891e56b2f5cbe606c8998d86f8eb535fc4bf9ecabf9"}
Oct 31 03:31:46.137243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:46.137213    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30" path="/var/lib/kubelet/pods/2eb8a5ac-7925-4e51-a5c5-2b6ce9151a30/volumes"
Oct 31 03:31:47.101312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:47.101274    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" event={"ID":"3b56c5a7-b345-46a9-bf9c-27a67c25a64b","Type":"ContainerStarted","Data":"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"}
Oct 31 03:31:47.101773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:47.101367    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:31:47.218053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:47.217977    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" podStartSLOduration=3.217953202 podStartE2EDuration="3.217953202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:47.213344883 +0000 UTC m=+1457.227906592" watchObservedRunningTime="2026-10-31 03:31:47.217953202 +0000 UTC m=+1457.232514919"
Oct 31 03:31:49.045914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:49.045880    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"]
Oct 31 03:31:49.091516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:49.091476    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b340484-fhmpn"]
Oct 31 03:31:50.137170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:50.137134    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b89131-2728-4288-ac7b-0c887f6a8a7e" path="/var/lib/kubelet/pods/35b89131-2728-4288-ac7b-0c887f6a8a7e/volumes"
Oct 31 03:31:51.046472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.046437    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"]
Oct 31 03:31:51.094067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.094039    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0952bd3f-tcxn7"]
Oct 31 03:31:51.331217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.331131    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"]
Oct 31 03:31:51.337172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.337153    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.385720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.385666    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:51.386028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.385933    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:31:51.386180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.385959    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:51.386311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.385973    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:51.390000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.389974    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"]
Oct 31 03:31:51.431935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.431901    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.432099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.431954    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp7z\" (UniqueName: \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.432145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.432120    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.533854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.533815    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp7z\" (UniqueName: \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.534041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.533959    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.534208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.534192    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.534840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.534817    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.536624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.536603    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.587597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.587479    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp7z\" (UniqueName: \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\") pod \"maas-api-revoke-sub-420a558d-wv4zq\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:51.651613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:51.651577    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:52.030407 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:31:52.030375    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231284c4_27b1_4e64_816a_bffe8c8c8f48.slice/crio-6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f WatchSource:0}: Error finding container 6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f: Status 404 returned error can't find the container with id 6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f
Oct 31 03:31:52.031663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:52.031638    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"]
Oct 31 03:31:52.138424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:52.138383    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c6b06a-753f-4922-bb7c-d66063c5a790" path="/var/lib/kubelet/pods/28c6b06a-753f-4922-bb7c-d66063c5a790/volumes"
Oct 31 03:31:52.147898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:52.147871    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq" event={"ID":"231284c4-27b1-4e64-816a-bffe8c8c8f48","Type":"ContainerStarted","Data":"6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f"}
Oct 31 03:31:53.158414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:53.158373    5831 generic.go:356] "Generic (PLEG): container finished" podID="231284c4-27b1-4e64-816a-bffe8c8c8f48" containerID="b062ab21068c5d04b23b1d638f28a8dc4d468b4faf0e5a3d4ba725890be7bb4f" exitCode=0
Oct 31 03:31:53.158825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:53.158458    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq" event={"ID":"231284c4-27b1-4e64-816a-bffe8c8c8f48","Type":"ContainerDied","Data":"b062ab21068c5d04b23b1d638f28a8dc4d468b4faf0e5a3d4ba725890be7bb4f"}
Oct 31 03:31:54.355224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.355197    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:54.377020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.376988    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\") pod \"231284c4-27b1-4e64-816a-bffe8c8c8f48\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") "
Oct 31 03:31:54.377171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.377083    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\" (UniqueName: \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\") pod \"231284c4-27b1-4e64-816a-bffe8c8c8f48\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") "
Oct 31 03:31:54.377370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.377350    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\") pod \"231284c4-27b1-4e64-816a-bffe8c8c8f48\" (UID: \"231284c4-27b1-4e64-816a-bffe8c8c8f48\") "
Oct 31 03:31:54.377429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.377366    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca" pod "231284c4-27b1-4e64-816a-bffe8c8c8f48" (UID: "231284c4-27b1-4e64-816a-bffe8c8c8f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:54.377978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.377924    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:54.379403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.379374    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z" pod "231284c4-27b1-4e64-816a-bffe8c8c8f48" (UID: "231284c4-27b1-4e64-816a-bffe8c8c8f48"). InnerVolumeSpecName "kube-api-access-7pp7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:54.379503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.379486    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca" pod "231284c4-27b1-4e64-816a-bffe8c8c8f48" (UID: "231284c4-27b1-4e64-816a-bffe8c8c8f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:54.479405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.479364    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231284c4-27b1-4e64-816a-bffe8c8c8f48-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:54.479405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:54.479408    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp7z\" (UniqueName: \"kubernetes.io/projected/231284c4-27b1-4e64-816a-bffe8c8c8f48-kube-api-access-7pp7z\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:55.078501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:55.078462    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:31:55.078701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:55.078569    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:31:55.178437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:55.178407    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"
Oct 31 03:31:55.178662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:55.178407    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq" event={"ID":"231284c4-27b1-4e64-816a-bffe8c8c8f48","Type":"ContainerDied","Data":"6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f"}
Oct 31 03:31:55.178662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:55.178554    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0f08dc9566ede0d5db9ec461f519fdcd145a1bce2824d9d7b8bc6d93e1c42f"
Oct 31 03:31:56.347856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:56.343029    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:31:58.115581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:31:58.115496    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:32:02.047025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:02.046988    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"]
Oct 31 03:32:02.107556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:02.104139    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a17f3c1-96pt4"]
Oct 31 03:32:02.137201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:02.137167    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12023e58-dc6a-407b-9a3d-50b07bc4f7f2" path="/var/lib/kubelet/pods/12023e58-dc6a-407b-9a3d-50b07bc4f7f2/volumes"
Oct 31 03:32:08.699154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.699117    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerID="695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6" exitCode=0
Oct 31 03:32:08.699651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.699195    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerDied","Data":"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"}
Oct 31 03:32:08.699703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.699692    5831 scope.go:122] "RemoveContainer" containerID="695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"
Oct 31 03:32:08.701588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.701556    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:08.701692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.701664    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:08.701742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:08.701719    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:09.726013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:09.725977    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerStarted","Data":"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"}
Oct 31 03:32:09.966156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:09.966125    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:32:09.967028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:09.967004    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:09.967161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:09.967061    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:10.966053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:10.966014    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:10.966621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:10.966080    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:11.021277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.021239    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:32:11.022463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.022439    5831 cpu_manager.go:414] "containerMap: removing container" podUID="231284c4-27b1-4e64-816a-bffe8c8c8f48" containerName="revoke-keys"
Oct 31 03:32:11.022463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.022463    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231284c4-27b1-4e64-816a-bffe8c8c8f48" containerName="revoke-keys"
Oct 31 03:32:11.022769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.022755    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="231284c4-27b1-4e64-816a-bffe8c8c8f48" containerName="revoke-keys"
Oct 31 03:32:11.027016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.026996    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.064202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.064172    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"]
Oct 31 03:32:11.068727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.068700    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3b8701-cbw2m"]
Oct 31 03:32:11.117247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.117215    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:32:11.143298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143306    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143336    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143356    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jqq\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143406    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143430    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143476    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143514    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.143595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.143557    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.244489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244400    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.244686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244588    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.244686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244629    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.244917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244898    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244933    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.244942    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245000    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245037    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jqq\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245170    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245221    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245282    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245328    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.245645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.245619    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.246165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.246107    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.247836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.247815    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.247926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.247818    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.302554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.300495    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.306564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.306343    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jqq\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.342272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.342243    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:11.820305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.820264    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:32:11.855544 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:32:11.855495    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e95ce24_ffa0_4529_a5d7_4001b0f7365e.slice/crio-50eaf0e80d64d5564493276acbd37f180041b68a1252f3500ca088d7d62b37fd WatchSource:0}: Error finding container 50eaf0e80d64d5564493276acbd37f180041b68a1252f3500ca088d7d62b37fd: Status 404 returned error can't find the container with id 50eaf0e80d64d5564493276acbd37f180041b68a1252f3500ca088d7d62b37fd
Oct 31 03:32:11.857806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.857768    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:11.857931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.857860    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:11.857931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.857908    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:11.966647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.966609    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:11.967103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:11.966669    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:12.140422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.140342    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14e2c2f-f733-4544-9b00-b8c97899245d" path="/var/lib/kubelet/pods/c14e2c2f-f733-4544-9b00-b8c97899245d/volumes"
Oct 31 03:32:12.755623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.755581    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" event={"ID":"9e95ce24-ffa0-4529-a5d7-4001b0f7365e","Type":"ContainerStarted","Data":"a68797c8f89fcec5c461f64d383afe8e5f4383df7e230e5115a94eb62554f3db"}
Oct 31 03:32:12.755840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.755638    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" event={"ID":"9e95ce24-ffa0-4529-a5d7-4001b0f7365e","Type":"ContainerStarted","Data":"50eaf0e80d64d5564493276acbd37f180041b68a1252f3500ca088d7d62b37fd"}
Oct 31 03:32:12.906557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.905584    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podStartSLOduration=2.905564724 podStartE2EDuration="2.905564724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:12.904102219 +0000 UTC m=+1482.918663926" watchObservedRunningTime="2026-10-31 03:32:12.905564724 +0000 UTC m=+1482.920126435"
Oct 31 03:32:12.976584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.976004    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:12.976584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:12.976106    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:13.214885 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:13.214851    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:32:13.349556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:13.348511    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:13.355024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:13.354977    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:13.355244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:13.355211    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:13.966625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:13.966574    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:13.966932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:13.966898    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:14.305612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:14.304520    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:32:14.343112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:14.343049    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:14.343285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:14.343143    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:14.966835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:14.966799    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 03:32:14.967159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:14.967124    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 03:32:15.347604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:15.344987    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:15.347604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:15.345055    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:15.971421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:15.971389    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:32:15.972742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:15.972719    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:32:16.346779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:16.346235    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:16.346779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:16.346312    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:17.343559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:17.343489    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:17.344083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:17.343589    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:18.066871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:18.066828    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"]
Oct 31 03:32:18.122555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:18.122136    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0069df22-bpmvs"]
Oct 31 03:32:18.177666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:18.177161    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8006bb6-9585-4e5d-a22f-30378af0645e" path="/var/lib/kubelet/pods/c8006bb6-9585-4e5d-a22f-30378af0645e/volumes"
Oct 31 03:32:18.343312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:18.343227    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:18.343312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:18.343284    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:19.044957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:19.044925    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"]
Oct 31 03:32:19.090476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:19.090444    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b5c7a4-g5gsc"]
Oct 31 03:32:19.342932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:19.342852    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 03:32:19.342932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:19.342910    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 03:32:20.141309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:20.141275    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12945d1f-09a7-434f-ac04-b9063b561410" path="/var/lib/kubelet/pods/12945d1f-09a7-434f-ac04-b9063b561410/volumes"
Oct 31 03:32:20.346708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:20.346681    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:20.346987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:20.346965    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:20.347651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:20.347635    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:32:24.046523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:24.046483    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"]
Oct 31 03:32:24.092104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:24.092068    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539ce198-zvthb"]
Oct 31 03:32:24.139023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:24.138984    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef423a2-bb90-4755-b816-ee14b9d953c6" path="/var/lib/kubelet/pods/5ef423a2-bb90-4755-b816-ee14b9d953c6/volumes"
Oct 31 03:32:33.372205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.372174    5831 scope.go:122] "RemoveContainer" containerID="9928b08645e111637afe900b6cf13bc40cb96eab844f3e667163cdf8fccd555f"
Oct 31 03:32:33.388341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.388293    5831 scope.go:122] "RemoveContainer" containerID="7a527a22e5d8d1e07eb9e4096149e3be1eb8553c9211ed04f4014edee513721a"
Oct 31 03:32:33.399745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.399723    5831 scope.go:122] "RemoveContainer" containerID="a75fbf052b7f9b1c21b2dd79729b70e664715aaa6862c80895edb4c9983f9548"
Oct 31 03:32:33.410733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.410711    5831 scope.go:122] "RemoveContainer" containerID="8c85b5e0b9e26d6f800d99737da5d737d80af3af1e471ef2d618179e2debb126"
Oct 31 03:32:33.420772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.420754    5831 scope.go:122] "RemoveContainer" containerID="b84c814da335f3d9de7ec676335c4b5726b48f32b4249594597a74257c91e2dd"
Oct 31 03:32:33.430794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.430733    5831 scope.go:122] "RemoveContainer" containerID="a8514f9062a3ada6e6c47e9b432da7d9e91f737907863c8acf1dcd186c2a14e4"
Oct 31 03:32:33.440938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:33.440917    5831 scope.go:122] "RemoveContainer" containerID="6be5dbbf299f7ef3093e7e2b6ec3761b091aa6337aa91253420e7176b6f7a9dc"
Oct 31 03:32:37.977756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:37.977731    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:32:37.989226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:37.989195    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:32:38.009950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:38.009912    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:32:38.022332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:38.022304    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:32:38.024626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:38.024600    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:32:38.026317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:38.026296    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:32:43.100790 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:43.100742    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:32:43.412605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:43.407276    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:32:46.067058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:46.067021    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:32:46.220547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:46.220495    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:32:49.329245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.329096    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:32:49.690931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.690864    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy" containerID="cri-o://f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db" gracePeriod=30
Oct 31 03:32:49.725329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.721818    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:32:49.726039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.725978    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" containerID="cri-o://dcf60afd41e9ba7bd2adad581603a3cea2d17cf6330b34dd53f18a71c18724c9" gracePeriod=30
Oct 31 03:32:49.733800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.733776    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:32:49.734247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:49.734219    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" containerID="cri-o://1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d" gracePeriod=30
Oct 31 03:32:51.068189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.068144    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" podUID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" containerName="payload-processing" containerID="cri-o://5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336" gracePeriod=25
Oct 31 03:32:51.228568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.226981    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" podUID="35bebe86-9191-4735-9073-5eb2f03eb07d" containerName="payload-pre-processing" containerID="cri-o://abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c" gracePeriod=25
Oct 31 03:32:51.349294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.349211    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"]
Oct 31 03:32:51.410805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.410777    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b3a29ec-tb7d6"]
Oct 31 03:32:51.579220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.578845    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:32:51.686221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.685925    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:32:51.759225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.759190    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\" (UniqueName: \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\") pod \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") "
Oct 31 03:32:51.759597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.759570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\") pod \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\" (UID: \"600a7d70-95aa-4747-b9ee-58f2f039cf7a\") "
Oct 31 03:32:51.760067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.760033    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume" pod "600a7d70-95aa-4747-b9ee-58f2f039cf7a" (UID: "600a7d70-95aa-4747-b9ee-58f2f039cf7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:51.760710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.760665    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600a7d70-95aa-4747-b9ee-58f2f039cf7a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:51.763239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.763218    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt" pod "600a7d70-95aa-4747-b9ee-58f2f039cf7a" (UID: "600a7d70-95aa-4747-b9ee-58f2f039cf7a"). InnerVolumeSpecName "kube-api-access-jzxwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:51.763969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.763943    5831 generic.go:356] "Generic (PLEG): container finished" podID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" containerID="5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336" exitCode=0
Oct 31 03:32:51.764171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.764151    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" event={"ID":"600a7d70-95aa-4747-b9ee-58f2f039cf7a","Type":"ContainerDied","Data":"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"}
Oct 31 03:32:51.764233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.764209    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn" event={"ID":"600a7d70-95aa-4747-b9ee-58f2f039cf7a","Type":"ContainerDied","Data":"a43b557e522e570b697498f6c1fc1ea29291c9d6260778f68e78a740f4a91a78"}
Oct 31 03:32:51.764281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.764240    5831 scope.go:122] "RemoveContainer" containerID="5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"
Oct 31 03:32:51.764424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.764407    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"
Oct 31 03:32:51.780017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.779987    5831 generic.go:356] "Generic (PLEG): container finished" podID="35bebe86-9191-4735-9073-5eb2f03eb07d" containerID="abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c" exitCode=0
Oct 31 03:32:51.780604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.780163    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" event={"ID":"35bebe86-9191-4735-9073-5eb2f03eb07d","Type":"ContainerDied","Data":"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"}
Oct 31 03:32:51.780604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.780220    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s" event={"ID":"35bebe86-9191-4735-9073-5eb2f03eb07d","Type":"ContainerDied","Data":"59f327a9f19b10a168e38924ec0064bcbe4812040761f02dda8bd080870ec5c5"}
Oct 31 03:32:51.780604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.780302    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"
Oct 31 03:32:51.787551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.787513    5831 scope.go:122] "RemoveContainer" containerID="5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"
Oct 31 03:32:51.787878 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:51.787852    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336\": container with ID starting with 5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336 not found: ID does not exist" containerID="5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"
Oct 31 03:32:51.787973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.787889    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336"} err="failed to get container status \"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336\": rpc error: code = NotFound desc = could not find container \"5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336\": container with ID starting with 5eaaf2200031810874862349b7809e284f96380b5c39d8c0157c38cabf654336 not found: ID does not exist"
Oct 31 03:32:51.787973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.787913    5831 scope.go:122] "RemoveContainer" containerID="abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"
Oct 31 03:32:51.806642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.806620    5831 scope.go:122] "RemoveContainer" containerID="abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"
Oct 31 03:32:51.807172 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:51.807146    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c\": container with ID starting with abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c not found: ID does not exist" containerID="abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"
Oct 31 03:32:51.807259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.807181    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c"} err="failed to get container status \"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c\": rpc error: code = NotFound desc = could not find container \"abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c\": container with ID starting with abe4ec1f0d88666990e1dddc7a4487722e00c8785b780f44c38018068bb07e9c not found: ID does not exist"
Oct 31 03:32:51.862349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.862316    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\") pod \"35bebe86-9191-4735-9073-5eb2f03eb07d\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") "
Oct 31 03:32:51.862562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.862461    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\" (UniqueName: \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\") pod \"35bebe86-9191-4735-9073-5eb2f03eb07d\" (UID: \"35bebe86-9191-4735-9073-5eb2f03eb07d\") "
Oct 31 03:32:51.862778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.862753    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume" pod "35bebe86-9191-4735-9073-5eb2f03eb07d" (UID: "35bebe86-9191-4735-9073-5eb2f03eb07d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:51.863117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.863096    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzxwt\" (UniqueName: \"kubernetes.io/projected/600a7d70-95aa-4747-b9ee-58f2f039cf7a-kube-api-access-jzxwt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:51.863117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.863117    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bebe86-9191-4735-9073-5eb2f03eb07d-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:51.865815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.865787    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp" pod "35bebe86-9191-4735-9073-5eb2f03eb07d" (UID: "35bebe86-9191-4735-9073-5eb2f03eb07d"). InnerVolumeSpecName "kube-api-access-5n9pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:51.951547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.951501    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:32:51.956442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.956410    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a16786-77487b7677-frnsn"]
Oct 31 03:32:51.964075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:51.964051    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9pp\" (UniqueName: \"kubernetes.io/projected/35bebe86-9191-4735-9073-5eb2f03eb07d-kube-api-access-5n9pp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:52.139446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:52.139417    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" path="/var/lib/kubelet/pods/600a7d70-95aa-4747-b9ee-58f2f039cf7a/volumes"
Oct 31 03:32:52.139972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:52.139956    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fd5ca8-e505-46fe-a582-acdc78d2e8a8" path="/var/lib/kubelet/pods/a5fd5ca8-e505-46fe-a582-acdc78d2e8a8/volumes"
Oct 31 03:32:52.200828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:52.200788    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:32:52.208729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:52.208652    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a16786-768455c76c-scw9s"]
Oct 31 03:32:53.961381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:53.961339    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:32:53.962121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:53.961411    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:32:54.139937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:54.139905    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bebe86-9191-4735-9073-5eb2f03eb07d" path="/var/lib/kubelet/pods/35bebe86-9191-4735-9073-5eb2f03eb07d/volumes"
Oct 31 03:32:54.858572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:54.858544    5831 generic.go:356] "Generic (PLEG): container finished" podID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerID="dcf60afd41e9ba7bd2adad581603a3cea2d17cf6330b34dd53f18a71c18724c9" exitCode=0
Oct 31 03:32:54.858711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:54.858610    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" event={"ID":"394efdbe-e6c8-4c26-9f74-01ae047efee5","Type":"ContainerDied","Data":"dcf60afd41e9ba7bd2adad581603a3cea2d17cf6330b34dd53f18a71c18724c9"}
Oct 31 03:32:55.201859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.201827    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:32:55.206038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.206004    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:32:55.206149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.206061    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329136    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329251    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329313    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329381    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329544    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.329634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329603    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.330097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329638    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.330097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329679    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.330097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.329821    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\") pod \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\" (UID: \"9c132719-ed85-457c-aeb8-2e5a2d49c15e\") "
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.331284    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.331355    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.331497    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.332469    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.332658    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.334589    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "kube-api-access-q6fbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.335566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.335502    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:32:55.354169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.335927    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.354169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.336952    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy" pod "9c132719-ed85-457c-aeb8-2e5a2d49c15e" (UID: "9c132719-ed85-457c-aeb8-2e5a2d49c15e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431788    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431824    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431840    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431853    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431865    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431877    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431889    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6fbg\" (UniqueName: \"kubernetes.io/projected/9c132719-ed85-457c-aeb8-2e5a2d49c15e-kube-api-access-q6fbg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431903    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.432079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.431919    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c132719-ed85-457c-aeb8-2e5a2d49c15e-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.505348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.500066    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:32:55.636021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.635987    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636084    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636178    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636203    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636336    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636368    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636403    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636440    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.636515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.636484    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\") pod \"394efdbe-e6c8-4c26-9f74-01ae047efee5\" (UID: \"394efdbe-e6c8-4c26-9f74-01ae047efee5\") "
Oct 31 03:32:55.640794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.640759    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "kube-api-access-qmxxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.641152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.641113    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.641347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.641327    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.641585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.641517    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.641929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.641897    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:55.646581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.643917    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:32:55.646581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.644328    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.647823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.647781    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.649381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.649351    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token" pod "394efdbe-e6c8-4c26-9f74-01ae047efee5" (UID: "394efdbe-e6c8-4c26-9f74-01ae047efee5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.661661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.661642    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767090    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767170    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767234    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767297    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767370    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767482    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767561    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767589    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.775602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.767623    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\") pod \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\" (UID: \"4be5abeb-6295-459f-94ed-6ad3da74b5b8\") "
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.801864    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803658    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803699    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394efdbe-e6c8-4c26-9f74-01ae047efee5-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803719    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxxp\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-kube-api-access-qmxxp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803746    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803769    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803791    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803811    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803823    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803837    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.803850    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394efdbe-e6c8-4c26-9f74-01ae047efee5-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.804071    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.804258    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.820187    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.846649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.828882    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:55.872618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.850134    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:55.872618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.850277    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.872618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.870668    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:32:55.873406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.873364    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r" pod "4be5abeb-6295-459f-94ed-6ad3da74b5b8" (UID: "4be5abeb-6295-459f-94ed-6ad3da74b5b8"). InnerVolumeSpecName "kube-api-access-z2l6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946076    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946115    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946129    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946144    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946158    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946171    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946187    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5abeb-6295-459f-94ed-6ad3da74b5b8-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.946552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.946200    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2l6r\" (UniqueName: \"kubernetes.io/projected/4be5abeb-6295-459f-94ed-6ad3da74b5b8-kube-api-access-z2l6r\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:55.951770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.948489    5831 generic.go:356] "Generic (PLEG): container finished" podID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerID="1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d" exitCode=0
Oct 31 03:32:55.951770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.948794    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" event={"ID":"9c132719-ed85-457c-aeb8-2e5a2d49c15e","Type":"ContainerDied","Data":"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"}
Oct 31 03:32:55.951770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.948924    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb" event={"ID":"9c132719-ed85-457c-aeb8-2e5a2d49c15e","Type":"ContainerDied","Data":"36d03662f0cc68e3074260dd7711fa6d0213ecda9b72f2d2f80f4a1edf4514cb"}
Oct 31 03:32:55.951770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.948958    5831 scope.go:122] "RemoveContainer" containerID="1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"
Oct 31 03:32:55.951770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.949201    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"
Oct 31 03:32:55.954007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.953973    5831 generic.go:356] "Generic (PLEG): container finished" podID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerID="f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db" exitCode=0
Oct 31 03:32:55.954128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.954074    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" event={"ID":"4be5abeb-6295-459f-94ed-6ad3da74b5b8","Type":"ContainerDied","Data":"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"}
Oct 31 03:32:55.954196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.954138    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw" event={"ID":"4be5abeb-6295-459f-94ed-6ad3da74b5b8","Type":"ContainerDied","Data":"f8f49e0fef800d85893ef9bf665856aa01377dd2269ab8a9daba1fb093c5081c"}
Oct 31 03:32:55.954254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.954240    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"
Oct 31 03:32:55.962883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.962854    5831 generic.go:356] "Generic (PLEG): container finished" podID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerID="e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a" exitCode=0
Oct 31 03:32:55.963170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.963142    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerDied","Data":"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"}
Oct 31 03:32:55.963800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.963780    5831 scope.go:122] "RemoveContainer" containerID="e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"
Oct 31 03:32:55.965341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.965316    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g" event={"ID":"394efdbe-e6c8-4c26-9f74-01ae047efee5","Type":"ContainerDied","Data":"2ca5cc3cf2d75a85c8478081811477de9625600901a52a290de6fb63a5252a7c"}
Oct 31 03:32:55.965431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.965414    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"
Oct 31 03:32:55.967157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.967137    5831 scope.go:122] "RemoveContainer" containerID="1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"
Oct 31 03:32:55.967698 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:55.967667    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d\": container with ID starting with 1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d not found: ID does not exist" containerID="1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"
Oct 31 03:32:55.967803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.967705    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d"} err="failed to get container status \"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d\": rpc error: code = NotFound desc = could not find container \"1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d\": container with ID starting with 1b4aa3549b5b61f66f5c9654c38f5d3d96b9ce022de74a550929b35fb0e8c61d not found: ID does not exist"
Oct 31 03:32:55.967803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.967729    5831 scope.go:122] "RemoveContainer" containerID="f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"
Oct 31 03:32:55.968331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.968302    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:55.968400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.968376    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:55.968465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:55.968407    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:32:56.007887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.007756    5831 scope.go:122] "RemoveContainer" containerID="f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"
Oct 31 03:32:56.008222 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:32:56.008193    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db\": container with ID starting with f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db not found: ID does not exist" containerID="f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"
Oct 31 03:32:56.008340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.008247    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db"} err="failed to get container status \"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db\": rpc error: code = NotFound desc = could not find container \"f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db\": container with ID starting with f5c3a6c65d3fde5bceef39b7e4408d3f03b15f57b05b775338b8c1cca8b7c2db not found: ID does not exist"
Oct 31 03:32:56.008340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.008270    5831 scope.go:122] "RemoveContainer" containerID="dcf60afd41e9ba7bd2adad581603a3cea2d17cf6330b34dd53f18a71c18724c9"
Oct 31 03:32:56.116652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.116610    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:32:56.119802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.119778    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xw77g"]
Oct 31 03:32:56.138040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.137997    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" path="/var/lib/kubelet/pods/394efdbe-e6c8-4c26-9f74-01ae047efee5/volumes"
Oct 31 03:32:56.220122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.220088    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:32:56.227020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.226992    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-qr9rb"]
Oct 31 03:32:56.429726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.429691    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:32:56.434518 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.434489    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-mkwhw"]
Oct 31 03:32:56.992598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:56.988835    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerStarted","Data":"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"}
Oct 31 03:32:57.034716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:57.034683    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:32:57.034944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:57.034919    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:32:57.035001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:57.034978    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:32:58.034758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:58.034722    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:32:58.035172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:58.034784    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:32:58.136710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:58.136671    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" path="/var/lib/kubelet/pods/4be5abeb-6295-459f-94ed-6ad3da74b5b8/volumes"
Oct 31 03:32:58.137252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:58.137238    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" path="/var/lib/kubelet/pods/9c132719-ed85-457c-aeb8-2e5a2d49c15e/volumes"
Oct 31 03:32:59.034794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:59.034717    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:32:59.035966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:32:59.034776    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:33:00.035332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:00.035300    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:33:00.035887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:00.035366    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:33:01.035402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:01.035364    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:33:01.035829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:01.035433    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:33:02.043983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:02.043943    5831 patch_prober.go:29] interesting pod/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 31 03:33:02.044600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:02.044564    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 31 03:33:03.039167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.039135    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:33:03.039681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.039660    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:33:03.040432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.040405    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:33:03.971600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.971561    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:33:03.973549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.973457    5831 cpu_manager.go:414] "containerMap: removing container" podUID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" containerName="payload-processing"
Oct 31 03:33:03.973874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.973854    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" containerName="payload-processing"
Oct 31 03:33:03.974029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974017    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy"
Oct 31 03:33:03.974104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974093    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy"
Oct 31 03:33:03.974189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974179    5831 cpu_manager.go:414] "containerMap: removing container" podUID="35bebe86-9191-4735-9073-5eb2f03eb07d" containerName="payload-pre-processing"
Oct 31 03:33:03.974273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974263    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bebe86-9191-4735-9073-5eb2f03eb07d" containerName="payload-pre-processing"
Oct 31 03:33:03.974357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974346    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy"
Oct 31 03:33:03.974442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974431    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy"
Oct 31 03:33:03.974633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974617    5831 cpu_manager.go:414] "containerMap: removing container" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy"
Oct 31 03:33:03.974763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.974750    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy"
Oct 31 03:33:03.975091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.975075    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="600a7d70-95aa-4747-b9ee-58f2f039cf7a" containerName="payload-processing"
Oct 31 03:33:03.975211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.975200    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c132719-ed85-457c-aeb8-2e5a2d49c15e" containerName="istio-proxy"
Oct 31 03:33:03.975290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.975281    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bebe86-9191-4735-9073-5eb2f03eb07d" containerName="payload-pre-processing"
Oct 31 03:33:03.975374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.975365    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be5abeb-6295-459f-94ed-6ad3da74b5b8" containerName="istio-proxy"
Oct 31 03:33:03.975463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:03.975454    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="394efdbe-e6c8-4c26-9f74-01ae047efee5" containerName="istio-proxy"
Oct 31 03:33:04.006776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.006742    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:33:04.007128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.007095    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.102557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.102507    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv57\" (UniqueName: \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.103044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.103017    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.179553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.147879    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b3a29ec\""
Oct 31 03:33:04.179553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.148666    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b3a29ec-dockercfg-wsl7r\""
Oct 31 03:33:04.219547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.219150    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv57\" (UniqueName: \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.219547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.219389    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.261123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.261034    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.282915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.282422    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv57\" (UniqueName: \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\") pod \"payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.415648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.415605    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:04.428321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.428275    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:33:04.436243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.436192    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:33:04.436413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.436321    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.607265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.607222    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.607414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.607333    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqhp\" (UniqueName: \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.709243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.709206    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.709441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.709275    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqhp\" (UniqueName: \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.710035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.710013    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.766019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.765984    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqhp\" (UniqueName: \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.790908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.790842    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:04.842064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:04.842021    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:33:04.843582 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:04.843548    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f4119d_604a_43cf_b9c7_0e9d4c52e086.slice/crio-f2514763b4b637a2c1b62a604b91201921f045e8932bc39ff30a0774000da2c7 WatchSource:0}: Error finding container f2514763b4b637a2c1b62a604b91201921f045e8932bc39ff30a0774000da2c7: Status 404 returned error can't find the container with id f2514763b4b637a2c1b62a604b91201921f045e8932bc39ff30a0774000da2c7
Oct 31 03:33:05.177201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.177172    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:33:05.178835 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:05.178807    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4089ad1f_4e4e_43a1_a4b1_a4bd257b028b.slice/crio-937211a927f848d5285fb466ee345d5f0d40cb3fcb103a51c5e94e66c093e762 WatchSource:0}: Error finding container 937211a927f848d5285fb466ee345d5f0d40cb3fcb103a51c5e94e66c093e762: Status 404 returned error can't find the container with id 937211a927f848d5285fb466ee345d5f0d40cb3fcb103a51c5e94e66c093e762
Oct 31 03:33:05.455850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.455810    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" event={"ID":"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b","Type":"ContainerStarted","Data":"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"}
Oct 31 03:33:05.456054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.455863    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" event={"ID":"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b","Type":"ContainerStarted","Data":"937211a927f848d5285fb466ee345d5f0d40cb3fcb103a51c5e94e66c093e762"}
Oct 31 03:33:05.456054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.455887    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:05.457593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.457564    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" event={"ID":"15f4119d-604a-43cf-b9c7-0e9d4c52e086","Type":"ContainerStarted","Data":"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"}
Oct 31 03:33:05.457719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.457616    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" event={"ID":"15f4119d-604a-43cf-b9c7-0e9d4c52e086","Type":"ContainerStarted","Data":"f2514763b4b637a2c1b62a604b91201921f045e8932bc39ff30a0774000da2c7"}
Oct 31 03:33:05.457719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.457641    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:05.565063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.565009    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" podStartSLOduration=1.56499572 podStartE2EDuration="1.56499572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:05.561328318 +0000 UTC m=+1535.575890026" watchObservedRunningTime="2026-10-31 03:33:05.56499572 +0000 UTC m=+1535.579557456"
Oct 31 03:33:05.699554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:05.695720    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" podStartSLOduration=2.695697185 podStartE2EDuration="2.695697185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:05.668445428 +0000 UTC m=+1535.683007136" watchObservedRunningTime="2026-10-31 03:33:05.695697185 +0000 UTC m=+1535.710258894"
Oct 31 03:33:06.045938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:06.045898    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"]
Oct 31 03:33:06.050350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:06.050326    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35eea44-rt2nv"]
Oct 31 03:33:06.136930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:06.136899    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614c321e-1034-4c7a-a78b-3f09183b05b2" path="/var/lib/kubelet/pods/614c321e-1034-4c7a-a78b-3f09183b05b2/volumes"
Oct 31 03:33:11.426076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.426042    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:33:11.426565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.426235    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy" containerID="cri-o://bc878ce8a8227ce834370a275b8a4ae5250bee5da920695237e4758d54c017e3" gracePeriod=30
Oct 31 03:33:11.472100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.472067    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:33:11.472302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.472284    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" containerID="cri-o://10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4" gracePeriod=30
Oct 31 03:33:11.477386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.477355    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:33:11.477728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.477678    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" containerID="cri-o://fa11c25d02db46e6b969fb1bd6c8b6dca15fa9dff926aa38dfbaa916eb5d0594" gracePeriod=30
Oct 31 03:33:11.516242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.516214    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:33:11.516495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:11.516470    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy" containerID="cri-o://251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b" gracePeriod=30
Oct 31 03:33:12.503631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:12.503576    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:33:13.060799 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:13.060755    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:33:15.831204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:15.831170    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:33:15.831653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:15.831227    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:33:16.317997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.317962    5831 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:33:16.318180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.318028    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:33:16.470056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.470031    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:33:16.470193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.470097    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:33:16.580129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.579774    5831 generic.go:356] "Generic (PLEG): container finished" podID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerID="fa11c25d02db46e6b969fb1bd6c8b6dca15fa9dff926aa38dfbaa916eb5d0594" exitCode=0
Oct 31 03:33:16.580129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.579878    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" event={"ID":"999063db-d9e8-4bdf-a237-ef3e008ad2db","Type":"ContainerDied","Data":"fa11c25d02db46e6b969fb1bd6c8b6dca15fa9dff926aa38dfbaa916eb5d0594"}
Oct 31 03:33:16.582647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.582449    5831 generic.go:356] "Generic (PLEG): container finished" podID="eafbce92-55cb-4bba-b666-504112c0cc11" containerID="bc878ce8a8227ce834370a275b8a4ae5250bee5da920695237e4758d54c017e3" exitCode=0
Oct 31 03:33:16.582647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.582590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" event={"ID":"eafbce92-55cb-4bba-b666-504112c0cc11","Type":"ContainerDied","Data":"bc878ce8a8227ce834370a275b8a4ae5250bee5da920695237e4758d54c017e3"}
Oct 31 03:33:16.840680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.840600    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:33:16.924591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.924371    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:33:16.932947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.932915    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933107    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933217    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933311    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933351    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933389    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933438    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933492    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.933654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.933576    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\") pod \"999063db-d9e8-4bdf-a237-ef3e008ad2db\" (UID: \"999063db-d9e8-4bdf-a237-ef3e008ad2db\") "
Oct 31 03:33:16.935161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.934911    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:16.935378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.935349    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.936509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.936245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.936868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.936695    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.937069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.937027    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.937434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.937407    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:16.943967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.943888    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:16.943967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.943928    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:16.946006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.945973    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg" pod "999063db-d9e8-4bdf-a237-ef3e008ad2db" (UID: "999063db-d9e8-4bdf-a237-ef3e008ad2db"). InnerVolumeSpecName "kube-api-access-scjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:16.962843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.962812    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:33:16.967417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:16.967392    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:33:17.035219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035181    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.035414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035291    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.035414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035369    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.035591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035403    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.035591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035456    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.035591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035495    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.035591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035502    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.035807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035627    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.035807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035699    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.035912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035887    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.036073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036040    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:17.036401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.035765    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.036507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036447    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.036507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036461    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.036507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036501    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.036770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.036770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036635    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.036770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036676    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.036770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036722    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.036770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036738    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036834    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036901    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036984    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\") pod \"eafbce92-55cb-4bba-b666-504112c0cc11\" (UID: \"eafbce92-55cb-4bba-b666-504112c0cc11\") "
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.036989    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.037014    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\") pod \"0030d86d-0aa5-4123-b0f6-340c2877e30a\" (UID: \"0030d86d-0aa5-4123-b0f6-340c2877e30a\") "
Oct 31 03:33:17.037645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.037264    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.038037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038009    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038039    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0030d86d-0aa5-4123-b0f6-340c2877e30a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038055    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038069    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999063db-d9e8-4bdf-a237-ef3e008ad2db-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038087    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038102    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038118    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038131    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038147    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999063db-d9e8-4bdf-a237-ef3e008ad2db-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038162    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038180    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjmg\" (UniqueName: \"kubernetes.io/projected/999063db-d9e8-4bdf-a237-ef3e008ad2db-kube-api-access-scjmg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038199    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafbce92-55cb-4bba-b666-504112c0cc11-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038215    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038232    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038247    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038262    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999063db-d9e8-4bdf-a237-ef3e008ad2db-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038279    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038499    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.038805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.038719    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.039199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.039169    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "kube-api-access-vkxsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.039317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.039245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.039630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.039604    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:17.039988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.039965    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "kube-api-access-jvxb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.040082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.040000    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token" pod "0030d86d-0aa5-4123-b0f6-340c2877e30a" (UID: "0030d86d-0aa5-4123-b0f6-340c2877e30a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.040580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.040559    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:17.040856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.040843    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy" pod "eafbce92-55cb-4bba-b666-504112c0cc11" (UID: "eafbce92-55cb-4bba-b666-504112c0cc11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.139447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139357    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139424    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139475    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139518    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139675    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139716    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.139876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139738    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.139876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139823    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.140043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139874    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.140043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.139923    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.140153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.140115    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\") pod \"18340bd6-ca06-4101-a97f-0f97f61e864a\" (UID: \"18340bd6-ca06-4101-a97f-0f97f61e864a\") "
Oct 31 03:33:17.140153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.140130    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:17.140569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.140386    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.140696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.140642    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.141146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141117    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141151    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141167    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18340bd6-ca06-4101-a97f-0f97f61e864a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141181    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafbce92-55cb-4bba-b666-504112c0cc11-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141195    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141209    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141224    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141237    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141251    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141264    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafbce92-55cb-4bba-b666-504112c0cc11-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141280    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxb2\" (UniqueName: \"kubernetes.io/projected/0030d86d-0aa5-4123-b0f6-340c2877e30a-kube-api-access-jvxb2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141294    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141308    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141322    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0030d86d-0aa5-4123-b0f6-340c2877e30a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.141686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.141335    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxsl\" (UniqueName: \"kubernetes.io/projected/eafbce92-55cb-4bba-b666-504112c0cc11-kube-api-access-vkxsl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.142842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.142811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:17.142930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.142874    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.143054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.143040    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "kube-api-access-qvtdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:17.143695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.143673    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo" pod "18340bd6-ca06-4101-a97f-0f97f61e864a" (UID: "18340bd6-ca06-4101-a97f-0f97f61e864a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:17.243135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.243093    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvtdm\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-kube-api-access-qvtdm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.243299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.243140    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.243299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.243157    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.243299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.243169    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18340bd6-ca06-4101-a97f-0f97f61e864a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:17.592862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.592829    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"
Oct 31 03:33:17.592862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.592841    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp" event={"ID":"eafbce92-55cb-4bba-b666-504112c0cc11","Type":"ContainerDied","Data":"bd4514fff83a69d9887fa189d89d720e3d9c051a0d03ef9d3c8360720f7d815a"}
Oct 31 03:33:17.593105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.592927    5831 scope.go:122] "RemoveContainer" containerID="bc878ce8a8227ce834370a275b8a4ae5250bee5da920695237e4758d54c017e3"
Oct 31 03:33:17.594708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.594682    5831 generic.go:356] "Generic (PLEG): container finished" podID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerID="10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4" exitCode=0
Oct 31 03:33:17.594827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.594738    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"
Oct 31 03:33:17.594885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.594812    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" event={"ID":"0030d86d-0aa5-4123-b0f6-340c2877e30a","Type":"ContainerDied","Data":"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"}
Oct 31 03:33:17.594885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.594866    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z" event={"ID":"0030d86d-0aa5-4123-b0f6-340c2877e30a","Type":"ContainerDied","Data":"75e1608b8978ee166bd79201f01070082da529d8d36057287a4673062890f61c"}
Oct 31 03:33:17.596844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.596816    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf" event={"ID":"999063db-d9e8-4bdf-a237-ef3e008ad2db","Type":"ContainerDied","Data":"89e5804d1108a1cb8fe46790fb013763c2fe3899cab8a22edcbd903e230ca63a"}
Oct 31 03:33:17.596844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.596843    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"
Oct 31 03:33:17.608251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.605546    5831 generic.go:356] "Generic (PLEG): container finished" podID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerID="251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b" exitCode=0
Oct 31 03:33:17.608251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.605682    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" event={"ID":"18340bd6-ca06-4101-a97f-0f97f61e864a","Type":"ContainerDied","Data":"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"}
Oct 31 03:33:17.608251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.605727    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm" event={"ID":"18340bd6-ca06-4101-a97f-0f97f61e864a","Type":"ContainerDied","Data":"e00f99efda1312101ff4c45493ebf94f0c3bcf6b4ee8d91678441b7192f13046"}
Oct 31 03:33:17.608251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.605818    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"
Oct 31 03:33:17.613918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.613900    5831 scope.go:122] "RemoveContainer" containerID="10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"
Oct 31 03:33:17.625792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.625744    5831 scope.go:122] "RemoveContainer" containerID="10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"
Oct 31 03:33:17.626138 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:17.626106    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4\": container with ID starting with 10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4 not found: ID does not exist" containerID="10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"
Oct 31 03:33:17.626226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.626146    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4"} err="failed to get container status \"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4\": rpc error: code = NotFound desc = could not find container \"10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4\": container with ID starting with 10f6eab5226153ac9a648cbd0ec11756b89628cbc4918b02d902f6b78b1f18a4 not found: ID does not exist"
Oct 31 03:33:17.626226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.626164    5831 scope.go:122] "RemoveContainer" containerID="fa11c25d02db46e6b969fb1bd6c8b6dca15fa9dff926aa38dfbaa916eb5d0594"
Oct 31 03:33:17.638203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.638172    5831 scope.go:122] "RemoveContainer" containerID="251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"
Oct 31 03:33:17.650063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.650046    5831 scope.go:122] "RemoveContainer" containerID="251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"
Oct 31 03:33:17.650372 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:17.650334    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b\": container with ID starting with 251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b not found: ID does not exist" containerID="251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"
Oct 31 03:33:17.650471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.650371    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b"} err="failed to get container status \"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b\": rpc error: code = NotFound desc = could not find container \"251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b\": container with ID starting with 251f1397b53bfe1da27a3607b3f8175ffe981f8d6e37d5778ea4d5047cb47d8b not found: ID does not exist"
Oct 31 03:33:17.749902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.749822    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:33:17.755899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.755872    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-lpp5z"]
Oct 31 03:33:17.879560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.875901    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:33:17.884562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.882821    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-hwpbp"]
Oct 31 03:33:17.975368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.975335    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:33:17.983438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:17.983409    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kq5hm"]
Oct 31 03:33:18.102195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.102116    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:33:18.108750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.108720    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-kfblf"]
Oct 31 03:33:18.136688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.136659    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" path="/var/lib/kubelet/pods/0030d86d-0aa5-4123-b0f6-340c2877e30a/volumes"
Oct 31 03:33:18.137165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.137152    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" path="/var/lib/kubelet/pods/18340bd6-ca06-4101-a97f-0f97f61e864a/volumes"
Oct 31 03:33:18.137599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.137586    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" path="/var/lib/kubelet/pods/999063db-d9e8-4bdf-a237-ef3e008ad2db/volumes"
Oct 31 03:33:18.138380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:18.138365    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" path="/var/lib/kubelet/pods/eafbce92-55cb-4bba-b666-504112c0cc11/volumes"
Oct 31 03:33:19.723129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.723099    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"]
Oct 31 03:33:19.724049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724026    5831 cpu_manager.go:414] "containerMap: removing container" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy"
Oct 31 03:33:19.724049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724050    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724067    5831 cpu_manager.go:414] "containerMap: removing container" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724075    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724124    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724132    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724142    5831 cpu_manager.go:414] "containerMap: removing container" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy"
Oct 31 03:33:19.724212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724149    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy"
Oct 31 03:33:19.724502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724343    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="999063db-d9e8-4bdf-a237-ef3e008ad2db" containerName="istio-proxy"
Oct 31 03:33:19.724502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724358    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafbce92-55cb-4bba-b666-504112c0cc11" containerName="istio-proxy"
Oct 31 03:33:19.724502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724376    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0030d86d-0aa5-4123-b0f6-340c2877e30a" containerName="istio-proxy"
Oct 31 03:33:19.724502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.724387    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="18340bd6-ca06-4101-a97f-0f97f61e864a" containerName="istio-proxy"
Oct 31 03:33:19.729826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.729804    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.784070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.784041    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"]
Oct 31 03:33:19.823664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.823628    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:33:19.823664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.823642    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:33:19.823910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.823688    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:33:19.827046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.827014    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:33:19.888369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.888338    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.888582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.888462    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5zw\" (UniqueName: \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.888673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.888656    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.990212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.990125    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.990212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.990206    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5zw\" (UniqueName: \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.990449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.990320    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.991008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.990981    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:19.992762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:19.992737    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:20.033164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:20.033127    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:33:20.033414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:20.033379    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy" containerID="cri-o://aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076" gracePeriod=30
Oct 31 03:33:20.045825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:20.045614    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5zw\" (UniqueName: \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\") pod \"maas-api-revoke-sub-52275d80-qjmqp\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:20.345889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:20.345791    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:20.779519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:20.779488    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"]
Oct 31 03:33:20.780239 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:20.780209    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090fedfe_0367_4015_aab1_5732719843d4.slice/crio-dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35 WatchSource:0}: Error finding container dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35: Status 404 returned error can't find the container with id dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35
Oct 31 03:33:21.668912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:21.668875    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp" event={"ID":"090fedfe-0367-4015-aab1-5732719843d4","Type":"ContainerStarted","Data":"53c11ce0ac2f52653b4767c31feb0cd997569d0757d8f224a7430db9f6ae8a8a"}
Oct 31 03:33:21.669097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:21.668927    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp" event={"ID":"090fedfe-0367-4015-aab1-5732719843d4","Type":"ContainerStarted","Data":"dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35"}
Oct 31 03:33:21.776266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:21.776209    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp" podStartSLOduration=2.776191372 podStartE2EDuration="2.776191372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:21.776069515 +0000 UTC m=+1551.790631224" watchObservedRunningTime="2026-10-31 03:33:21.776191372 +0000 UTC m=+1551.790753080"
Oct 31 03:33:22.047757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.047724    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"]
Oct 31 03:33:22.051897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.051868    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-267c6524-krc5z"]
Oct 31 03:33:22.136166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.136131    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23738506-c912-4b44-b007-04351abad98d" path="/var/lib/kubelet/pods/23738506-c912-4b44-b007-04351abad98d/volumes"
Oct 31 03:33:22.521066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.521022    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:33:22.521254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.521229    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy" containerID="cri-o://a1ab1b1cae713581ab7bb6057f9b6565e141560fcbc31ad43427770cfd2303df" gracePeriod=30
Oct 31 03:33:22.569588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.569554    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:33:22.569791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.569763    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" containerID="cri-o://c2f4ea80085703cd0697b2703a01abbdcd9dfc9877e0a1c6d0b0415b49c191ad" gracePeriod=30
Oct 31 03:33:22.574367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.574341    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:33:22.574615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.574592    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy" containerID="cri-o://19bf18d93e2405f22a6527ba43b4ee3239f51565e6570af85922aa0b8bb09c8e" gracePeriod=30
Oct 31 03:33:22.578468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.578445    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:33:22.578667 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.578646    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy" containerID="cri-o://35c058db943e2804d71615c8c5f2e6060e02060ccc7b1bf1e8f87c9e98bd2dfd" gracePeriod=30
Oct 31 03:33:22.680504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.680473    5831 generic.go:356] "Generic (PLEG): container finished" podID="090fedfe-0367-4015-aab1-5732719843d4" containerID="53c11ce0ac2f52653b4767c31feb0cd997569d0757d8f224a7430db9f6ae8a8a" exitCode=0
Oct 31 03:33:22.680684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:22.680590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp" event={"ID":"090fedfe-0367-4015-aab1-5732719843d4","Type":"ContainerDied","Data":"53c11ce0ac2f52653b4767c31feb0cd997569d0757d8f224a7430db9f6ae8a8a"}
Oct 31 03:33:23.903098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.903054    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:23.974072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.973978    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\") pod \"090fedfe-0367-4015-aab1-5732719843d4\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") "
Oct 31 03:33:23.974217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.974149    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\" (UniqueName: \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\") pod \"090fedfe-0367-4015-aab1-5732719843d4\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") "
Oct 31 03:33:23.974217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.974170    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\") pod \"090fedfe-0367-4015-aab1-5732719843d4\" (UID: \"090fedfe-0367-4015-aab1-5732719843d4\") "
Oct 31 03:33:23.974658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.974628    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca" pod "090fedfe-0367-4015-aab1-5732719843d4" (UID: "090fedfe-0367-4015-aab1-5732719843d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:23.974796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.974769    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/090fedfe-0367-4015-aab1-5732719843d4-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:23.977399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.977364    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca" pod "090fedfe-0367-4015-aab1-5732719843d4" (UID: "090fedfe-0367-4015-aab1-5732719843d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:23.977489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:23.977404    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw" pod "090fedfe-0367-4015-aab1-5732719843d4" (UID: "090fedfe-0367-4015-aab1-5732719843d4"). InnerVolumeSpecName "kube-api-access-vc5zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:24.083566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:24.083425    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc5zw\" (UniqueName: \"kubernetes.io/projected/090fedfe-0367-4015-aab1-5732719843d4-kube-api-access-vc5zw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:24.083566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:24.083467    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/090fedfe-0367-4015-aab1-5732719843d4-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:24.707089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:24.707055    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"
Oct 31 03:33:24.707368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:24.707081    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp" event={"ID":"090fedfe-0367-4015-aab1-5732719843d4","Type":"ContainerDied","Data":"dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35"}
Oct 31 03:33:24.707480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:24.707389    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5424910964bb121095b86274a8520627e965ff272886fd43329fd35624fc35"
Oct 31 03:33:25.348799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.348775    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:33:25.446771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.446740    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.446797    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.446856    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.446960    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447026    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447056    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447084    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:25.447203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447147    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:25.447435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447283    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:25.447435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447292    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447419    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:25.447680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447479    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\") pod \"39127796-4e80-49c5-ab27-950828b2c41a\" (UID: \"39127796-4e80-49c5-ab27-950828b2c41a\") "
Oct 31 03:33:25.447680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.447514    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:25.448465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.448438    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.448588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.448469    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.448588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.448484    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.448588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.448499    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39127796-4e80-49c5-ab27-950828b2c41a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.448588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.448513    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.449470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.449439    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:25.449470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.449452    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:25.449649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.449559    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "kube-api-access-7js9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:25.449708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.449693    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy" pod "39127796-4e80-49c5-ab27-950828b2c41a" (UID: "39127796-4e80-49c5-ab27-950828b2c41a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:25.549934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.549901    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39127796-4e80-49c5-ab27-950828b2c41a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.549934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.549934    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39127796-4e80-49c5-ab27-950828b2c41a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.550143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.549944    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.550143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.549953    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7js9w\" (UniqueName: \"kubernetes.io/projected/39127796-4e80-49c5-ab27-950828b2c41a-kube-api-access-7js9w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:25.717389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.717353    5831 generic.go:356] "Generic (PLEG): container finished" podID="39127796-4e80-49c5-ab27-950828b2c41a" containerID="aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076" exitCode=0
Oct 31 03:33:25.717592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.717397    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"
Oct 31 03:33:25.717592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.717426    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerDied","Data":"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"}
Oct 31 03:33:25.717592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.717471    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq" event={"ID":"39127796-4e80-49c5-ab27-950828b2c41a","Type":"ContainerDied","Data":"aed3acc8c7ff8663869ae2171cc0a576c88cf81038abb0d9c58cd6a0a8816481"}
Oct 31 03:33:25.717592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.717495    5831 scope.go:122] "RemoveContainer" containerID="aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"
Oct 31 03:33:25.729779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.729759    5831 scope.go:122] "RemoveContainer" containerID="c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"
Oct 31 03:33:25.740387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.740369    5831 scope.go:122] "RemoveContainer" containerID="aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"
Oct 31 03:33:25.740729 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:25.740704    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076\": container with ID starting with aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076 not found: ID does not exist" containerID="aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"
Oct 31 03:33:25.740825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.740742    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076"} err="failed to get container status \"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076\": rpc error: code = NotFound desc = could not find container \"aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076\": container with ID starting with aeb80101366eb8eae48ab78746180e3d36cb361f5985d92b193cab21ce766076 not found: ID does not exist"
Oct 31 03:33:25.740825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.740768    5831 scope.go:122] "RemoveContainer" containerID="c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"
Oct 31 03:33:25.741053 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:25.741035    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38\": container with ID starting with c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38 not found: ID does not exist" containerID="c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"
Oct 31 03:33:25.741099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.741060    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38"} err="failed to get container status \"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38\": rpc error: code = NotFound desc = could not find container \"c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38\": container with ID starting with c7f2d1f631327db9f317406cc3506b82648bab3e4dc978c7dfc2e9829af08b38 not found: ID does not exist"
Oct 31 03:33:25.873724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.873685    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:33:25.877726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:25.877692    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a16786-openshift-default-7fcd4876c-l7zxq"]
Oct 31 03:33:26.138668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:26.138591    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39127796-4e80-49c5-ab27-950828b2c41a" path="/var/lib/kubelet/pods/39127796-4e80-49c5-ab27-950828b2c41a/volumes"
Oct 31 03:33:26.250930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:26.250899    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:33:26.251096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:26.250948    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:33:27.749313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.749275    5831 generic.go:356] "Generic (PLEG): container finished" podID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerID="c2f4ea80085703cd0697b2703a01abbdcd9dfc9877e0a1c6d0b0415b49c191ad" exitCode=0
Oct 31 03:33:27.749799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.749442    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" event={"ID":"148e5fba-3872-40e2-9922-ea8e1de83ad4","Type":"ContainerDied","Data":"c2f4ea80085703cd0697b2703a01abbdcd9dfc9877e0a1c6d0b0415b49c191ad"}
Oct 31 03:33:27.752008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.751982    5831 generic.go:356] "Generic (PLEG): container finished" podID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerID="a1ab1b1cae713581ab7bb6057f9b6565e141560fcbc31ad43427770cfd2303df" exitCode=0
Oct 31 03:33:27.752165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.752067    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" event={"ID":"163cc81a-c7ed-47fc-adc0-125405ec8113","Type":"ContainerDied","Data":"a1ab1b1cae713581ab7bb6057f9b6565e141560fcbc31ad43427770cfd2303df"}
Oct 31 03:33:27.753661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.753635    5831 generic.go:356] "Generic (PLEG): container finished" podID="51c25628-063f-4996-b147-8210e12ce7fb" containerID="19bf18d93e2405f22a6527ba43b4ee3239f51565e6570af85922aa0b8bb09c8e" exitCode=0
Oct 31 03:33:27.753791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.753692    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" event={"ID":"51c25628-063f-4996-b147-8210e12ce7fb","Type":"ContainerDied","Data":"19bf18d93e2405f22a6527ba43b4ee3239f51565e6570af85922aa0b8bb09c8e"}
Oct 31 03:33:27.758632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.758597    5831 generic.go:356] "Generic (PLEG): container finished" podID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerID="35c058db943e2804d71615c8c5f2e6060e02060ccc7b1bf1e8f87c9e98bd2dfd" exitCode=0
Oct 31 03:33:27.758765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.758718    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerDied","Data":"35c058db943e2804d71615c8c5f2e6060e02060ccc7b1bf1e8f87c9e98bd2dfd"}
Oct 31 03:33:27.758836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.758768    5831 scope.go:122] "RemoveContainer" containerID="dfad3a5cce56281a765506388af29575945554fd956b6f036236f01090f01563"
Oct 31 03:33:27.922433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.922405    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:33:27.986813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.986783    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.986955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.986851    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.986955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.986894    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.986977    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987064    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.987202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987152    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987200    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987296    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987304    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.987463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987447    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.987570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.987492    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\") pod \"148e5fba-3872-40e2-9922-ea8e1de83ad4\" (UID: \"148e5fba-3872-40e2-9922-ea8e1de83ad4\") "
Oct 31 03:33:27.988245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.988220    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.988245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.988247    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.988804 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.988605    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:27.989031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.988892    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.989326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.989173    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.991456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.991420    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.993065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.993025    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "kube-api-access-qnt78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:27.993734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.993697    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:27.994234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:27.994211    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token" pod "148e5fba-3872-40e2-9922-ea8e1de83ad4" (UID: "148e5fba-3872-40e2-9922-ea8e1de83ad4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.009985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.009913    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:33:28.013138 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.013076    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:33:28.017873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.017854    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:33:28.090610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.090456    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.095758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.095622    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.096125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096099    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096561    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096596    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096620    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096665    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.096778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.096706    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.097151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097008    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097326    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097399    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097509    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097577    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097629    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097679    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097712    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097326    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097556    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097738    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097760    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.097918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.097844    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098102    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\") pod \"7dc49a73-12ea-40a4-afbc-08311509dd58\" (UID: \"7dc49a73-12ea-40a4-afbc-08311509dd58\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098147    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098197    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098281    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098334    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098381    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098420    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098494    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\") pod \"163cc81a-c7ed-47fc-adc0-125405ec8113\" (UID: \"163cc81a-c7ed-47fc-adc0-125405ec8113\") "
Oct 31 03:33:28.098849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.098522    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\") pod \"51c25628-063f-4996-b147-8210e12ce7fb\" (UID: \"51c25628-063f-4996-b147-8210e12ce7fb\") "
Oct 31 03:33:28.099643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.099364    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100075    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100104    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100119    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100132    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100147    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148e5fba-3872-40e2-9922-ea8e1de83ad4-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100161    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100175    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100189    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100202    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100216    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100230    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148e5fba-3872-40e2-9922-ea8e1de83ad4-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100243    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148e5fba-3872-40e2-9922-ea8e1de83ad4-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100257    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnt78\" (UniqueName: \"kubernetes.io/projected/148e5fba-3872-40e2-9922-ea8e1de83ad4-kube-api-access-qnt78\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.100585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.100428    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.101741    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.101866    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.102042    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.102072    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.102383    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.102686    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.103055    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:28.103373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.103272    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.104310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.104259    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.104419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.104328    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:28.105715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.105659    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.106958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.106901    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "kube-api-access-94mmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.107488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.107459    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.109904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.108048    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.109904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.108161    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:28.109904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.108273    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.109904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.108366    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo" pod "7dc49a73-12ea-40a4-afbc-08311509dd58" (UID: "7dc49a73-12ea-40a4-afbc-08311509dd58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:28.109904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.108511    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "kube-api-access-n9d9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.110937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.110904    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7" pod "163cc81a-c7ed-47fc-adc0-125405ec8113" (UID: "163cc81a-c7ed-47fc-adc0-125405ec8113"). InnerVolumeSpecName "kube-api-access-b4bv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:28.111252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.111220    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy" pod "51c25628-063f-4996-b147-8210e12ce7fb" (UID: "51c25628-063f-4996-b147-8210e12ce7fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:28.211202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211170    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163cc81a-c7ed-47fc-adc0-125405ec8113-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211441    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211479    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211494    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211507    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211521    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bv7\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-kube-api-access-b4bv7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211553    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c25628-063f-4996-b147-8210e12ce7fb-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211566    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211580    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211593    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211605    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211624    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mmr\" (UniqueName: \"kubernetes.io/projected/7dc49a73-12ea-40a4-afbc-08311509dd58-kube-api-access-94mmr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211636    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211648    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc49a73-12ea-40a4-afbc-08311509dd58-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211660    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc49a73-12ea-40a4-afbc-08311509dd58-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211672    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9d9d\" (UniqueName: \"kubernetes.io/projected/51c25628-063f-4996-b147-8210e12ce7fb-kube-api-access-n9d9d\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211684    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163cc81a-c7ed-47fc-adc0-125405ec8113-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211700    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c25628-063f-4996-b147-8210e12ce7fb-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211715    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.211723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211727    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c25628-063f-4996-b147-8210e12ce7fb-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.216231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.211745    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163cc81a-c7ed-47fc-adc0-125405ec8113-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.768024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.767999    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"
Oct 31 03:33:28.768464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.767995    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls" event={"ID":"163cc81a-c7ed-47fc-adc0-125405ec8113","Type":"ContainerDied","Data":"325daf83c2fc5359db98307f3ff078d9084ab1d52400e9426843aa41ccc6c3d8"}
Oct 31 03:33:28.768464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.768170    5831 scope.go:122] "RemoveContainer" containerID="a1ab1b1cae713581ab7bb6057f9b6565e141560fcbc31ad43427770cfd2303df"
Oct 31 03:33:28.769647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.769621    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"
Oct 31 03:33:28.769778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.769639    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs" event={"ID":"51c25628-063f-4996-b147-8210e12ce7fb","Type":"ContainerDied","Data":"4832af33a73c971d40ecfe5f986983b4ee92f4de1912b1dcb693d6b03175541b"}
Oct 31 03:33:28.773160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.771742    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"
Oct 31 03:33:28.773160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.771897    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl" event={"ID":"7dc49a73-12ea-40a4-afbc-08311509dd58","Type":"ContainerDied","Data":"c1db455f7656a14e1b74b2454a66503610fb77efa784eb632c6d7dc88f36a71e"}
Oct 31 03:33:28.773618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.773594    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn" event={"ID":"148e5fba-3872-40e2-9922-ea8e1de83ad4","Type":"ContainerDied","Data":"5928b8036f1945affb4bb6e90c1bd76c9c2a1b54ea8e128d316a8a2f53ec3953"}
Oct 31 03:33:28.773758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.773636    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"
Oct 31 03:33:28.783284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.783266    5831 scope.go:122] "RemoveContainer" containerID="19bf18d93e2405f22a6527ba43b4ee3239f51565e6570af85922aa0b8bb09c8e"
Oct 31 03:33:28.794352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.794329    5831 scope.go:122] "RemoveContainer" containerID="35c058db943e2804d71615c8c5f2e6060e02060ccc7b1bf1e8f87c9e98bd2dfd"
Oct 31 03:33:28.807943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.807922    5831 scope.go:122] "RemoveContainer" containerID="c2f4ea80085703cd0697b2703a01abbdcd9dfc9877e0a1c6d0b0415b49c191ad"
Oct 31 03:33:28.925216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.925179    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:33:28.930694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:28.930663    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-rfhls"]
Oct 31 03:33:29.072627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.070082    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:33:29.085554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.080637    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-vlssn"]
Oct 31 03:33:29.164107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.164064    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:33:29.169713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.169689    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-tskqs"]
Oct 31 03:33:29.273092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.273059    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:33:29.277050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:29.277029    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-p7xhl"]
Oct 31 03:33:30.223972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.223920    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" path="/var/lib/kubelet/pods/148e5fba-3872-40e2-9922-ea8e1de83ad4/volumes"
Oct 31 03:33:30.225570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.225498    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" path="/var/lib/kubelet/pods/163cc81a-c7ed-47fc-adc0-125405ec8113/volumes"
Oct 31 03:33:30.226403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.226381    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c25628-063f-4996-b147-8210e12ce7fb" path="/var/lib/kubelet/pods/51c25628-063f-4996-b147-8210e12ce7fb/volumes"
Oct 31 03:33:30.227604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.227583    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" path="/var/lib/kubelet/pods/7dc49a73-12ea-40a4-afbc-08311509dd58/volumes"
Oct 31 03:33:30.666332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.666261    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:33:30.717489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:30.717458    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:33:33.524318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.524290    5831 scope.go:122] "RemoveContainer" containerID="d829a2b4611545d0f6bd8e5d9f9137610d82e4497e8325e584d445ff2761ba97"
Oct 31 03:33:33.535850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.535830    5831 scope.go:122] "RemoveContainer" containerID="b795d6686ecc078f2fb9028d3d04f267952f8d10d2b4a257b0a9e4de7b3c44ca"
Oct 31 03:33:33.546119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.546099    5831 scope.go:122] "RemoveContainer" containerID="dbccf2baa805f6b077d406863c2ee384074dba96bdfa1670df993679aef96a8a"
Oct 31 03:33:33.556288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.556264    5831 scope.go:122] "RemoveContainer" containerID="3c9a7298834963829ef1b3b07b4a6820a0804324701e8625ee102f4f4ad3938c"
Oct 31 03:33:33.566959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.566940    5831 scope.go:122] "RemoveContainer" containerID="4ba5c36af420d3a16bec5582a6ea6f10a2d3125f061f11ab888745d5e0524adf"
Oct 31 03:33:33.577825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.577804    5831 scope.go:122] "RemoveContainer" containerID="29b6dfdbe195481988df5f7ac44355b904bd69c5e71968cac6f187eca8397fb7"
Oct 31 03:33:33.588353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.588318    5831 scope.go:122] "RemoveContainer" containerID="833dfea4c40a3a087d69fcc6efd7d5902cc77db911da0adfe3ea786ccc330f89"
Oct 31 03:33:33.599814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:33.599795    5831 scope.go:122] "RemoveContainer" containerID="e61928e793f11215259e6de5bcdc3c9eda389d95a186c0dad064c7881f702032"
Oct 31 03:33:35.666887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.666847    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" podUID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" containerName="payload-processing" containerID="cri-o://9b851b4675a4b1ebf2ed5bc182803365a539a16c4c6b5cd84eace3f97a95818d" gracePeriod=25
Oct 31 03:33:35.718699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.718648    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" podUID="4b7822da-6223-4e38-9216-cef279a067c9" containerName="payload-pre-processing" containerID="cri-o://8f2bf77b6c3c8f7987759cc448e7718c80b56df5504e548bb7a5e30255a24357" gracePeriod=25
Oct 31 03:33:35.921522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.921168    5831 generic.go:356] "Generic (PLEG): container finished" podID="4b7822da-6223-4e38-9216-cef279a067c9" containerID="8f2bf77b6c3c8f7987759cc448e7718c80b56df5504e548bb7a5e30255a24357" exitCode=0
Oct 31 03:33:35.921522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.921361    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" event={"ID":"4b7822da-6223-4e38-9216-cef279a067c9","Type":"ContainerDied","Data":"8f2bf77b6c3c8f7987759cc448e7718c80b56df5504e548bb7a5e30255a24357"}
Oct 31 03:33:35.926765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.926576    5831 generic.go:356] "Generic (PLEG): container finished" podID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" containerID="9b851b4675a4b1ebf2ed5bc182803365a539a16c4c6b5cd84eace3f97a95818d" exitCode=0
Oct 31 03:33:35.926765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:35.926698    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" event={"ID":"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a","Type":"ContainerDied","Data":"9b851b4675a4b1ebf2ed5bc182803365a539a16c4c6b5cd84eace3f97a95818d"}
Oct 31 03:33:36.153741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.153695    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202564    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202595    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202615    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202623    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202674    5831 cpu_manager.go:414] "containerMap: removing container" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202683    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202695    5831 cpu_manager.go:414] "containerMap: removing container" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.202703    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205053    5831 cpu_manager.go:414] "containerMap: removing container" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205075    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205123    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205131    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205146    5831 cpu_manager.go:414] "containerMap: removing container" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205158    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205183    5831 cpu_manager.go:414] "containerMap: removing container" podUID="090fedfe-0367-4015-aab1-5732719843d4" containerName="revoke-keys"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205191    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090fedfe-0367-4015-aab1-5732719843d4" containerName="revoke-keys"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205601    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c25628-063f-4996-b147-8210e12ce7fb" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205620    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="163cc81a-c7ed-47fc-adc0-125405ec8113" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205638    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="090fedfe-0367-4015-aab1-5732719843d4" containerName="revoke-keys"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205671    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205680    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc49a73-12ea-40a4-afbc-08311509dd58" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205694    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205712    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="39127796-4e80-49c5-ab27-950828b2c41a" containerName="istio-proxy"
Oct 31 03:33:36.208591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.205736    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="148e5fba-3872-40e2-9922-ea8e1de83ad4" containerName="istio-proxy"
Oct 31 03:33:36.231059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.231013    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.232221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.232202    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:33:36.251308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.251241    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:33:36.418900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.363345    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\" (UniqueName: \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\") pod \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") "
Oct 31 03:33:36.418900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.363392    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\") pod \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\" (UID: \"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a\") "
Oct 31 03:33:36.418900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.375196    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-87af9c-openshift-default-dockercfg-4dt4d\""
Oct 31 03:33:36.418900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.402842    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume" pod "a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" (UID: "a65f3b0d-d812-4b9a-9c42-78d8cf54e95a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:36.420331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.420299    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.420597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.420577    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.421194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.421174    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.430605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.421359    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.431240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.430970    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.431460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.431442    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.439498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.439455    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.439708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.439691    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5rc\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.456484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.456454    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.459821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.459754    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:36.571746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.571689    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt" pod "a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" (UID: "a65f3b0d-d812-4b9a-9c42-78d8cf54e95a"). InnerVolumeSpecName "kube-api-access-7mhrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:36.584951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.584915    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585190    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5rc\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585243    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585363    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585390    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585466    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585496    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585546    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585579    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.586578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.585732    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhrt\" (UniqueName: \"kubernetes.io/projected/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a-kube-api-access-7mhrt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:36.604720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.604684    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.605496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.605468    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.605991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.605967    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.606388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.606368    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.606882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.606861    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.611950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.611931    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:33:36.649868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.649782    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.678894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.678856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.692193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.692158    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.724465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.724433    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5rc\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\") pod \"e2e-api-b-87af9c-openshift-default-85d666bc67-p799k\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") " pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.802571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.800502    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\" (UniqueName: \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\") pod \"4b7822da-6223-4e38-9216-cef279a067c9\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") "
Oct 31 03:33:36.802571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.800631    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\") pod \"4b7822da-6223-4e38-9216-cef279a067c9\" (UID: \"4b7822da-6223-4e38-9216-cef279a067c9\") "
Oct 31 03:33:36.802571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.801269    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume" pod "4b7822da-6223-4e38-9216-cef279a067c9" (UID: "4b7822da-6223-4e38-9216-cef279a067c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:36.810231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.810186    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl" pod "4b7822da-6223-4e38-9216-cef279a067c9" (UID: "4b7822da-6223-4e38-9216-cef279a067c9"). InnerVolumeSpecName "kube-api-access-qgcgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:36.904569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.902780    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcgl\" (UniqueName: \"kubernetes.io/projected/4b7822da-6223-4e38-9216-cef279a067c9-kube-api-access-qgcgl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:36.904569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.902831    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7822da-6223-4e38-9216-cef279a067c9-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:36.926636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.922840    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:36.971182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.961572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb" event={"ID":"4b7822da-6223-4e38-9216-cef279a067c9","Type":"ContainerDied","Data":"762607cf6dbb63015c40d711a426b694b48dd95a0481a62881ecc8f9224e2c3b"}
Oct 31 03:33:36.971182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.961644    5831 scope.go:122] "RemoveContainer" containerID="8f2bf77b6c3c8f7987759cc448e7718c80b56df5504e548bb7a5e30255a24357"
Oct 31 03:33:36.971182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.961828    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"
Oct 31 03:33:36.975298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.974375    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m" event={"ID":"a65f3b0d-d812-4b9a-9c42-78d8cf54e95a","Type":"ContainerDied","Data":"a8536dd791d6ef2943e996677d8375a164de73f2a87a4de41913c2c805c7b790"}
Oct 31 03:33:36.975298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:36.974483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"
Oct 31 03:33:37.153440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.153416    5831 scope.go:122] "RemoveContainer" containerID="9b851b4675a4b1ebf2ed5bc182803365a539a16c4c6b5cd84eace3f97a95818d"
Oct 31 03:33:37.168085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.168054    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:33:37.356588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.347078    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:33:37.398124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.397808    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-070d07-7f49b754c5-7crbb"]
Oct 31 03:33:37.462562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.450048    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:33:37.631031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.630997    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:33:37.999634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:37.999588    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-070d07-6487b76589-zkd5m"]
Oct 31 03:33:38.082352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.081629    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:38.336554 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:38.331657    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6925218a_23f2_40b0_8762_c6adc4632afa.slice/crio-f4e241068d3d8dcf3c3d4b67d3a554ed9661a07ff92055ad47236cc0026983ea WatchSource:0}: Error finding container f4e241068d3d8dcf3c3d4b67d3a554ed9661a07ff92055ad47236cc0026983ea: Status 404 returned error can't find the container with id f4e241068d3d8dcf3c3d4b67d3a554ed9661a07ff92055ad47236cc0026983ea
Oct 31 03:33:38.363569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.355467    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:33:38.363569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.355597    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:33:38.363569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.355651    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:33:38.445556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.388159    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7822da-6223-4e38-9216-cef279a067c9" path="/var/lib/kubelet/pods/4b7822da-6223-4e38-9216-cef279a067c9/volumes"
Oct 31 03:33:38.449451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.448879    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" path="/var/lib/kubelet/pods/a65f3b0d-d812-4b9a-9c42-78d8cf54e95a/volumes"
Oct 31 03:33:38.449451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:38.449353    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:33:39.628554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:39.627141    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerStarted","Data":"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"}
Oct 31 03:33:39.628554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:39.627212    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerStarted","Data":"f4e241068d3d8dcf3c3d4b67d3a554ed9661a07ff92055ad47236cc0026983ea"}
Oct 31 03:33:39.875557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:39.875094    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podStartSLOduration=3.875069705 podStartE2EDuration="3.875069705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:39.850768052 +0000 UTC m=+1569.865329765" watchObservedRunningTime="2026-10-31 03:33:39.875069705 +0000 UTC m=+1569.889631414"
Oct 31 03:33:39.935555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:39.931502    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:39.951557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:39.949875    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:40.673284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:40.673252    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:40.674515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:40.674491    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:33:41.883878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:41.883830    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:33:41.884362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:41.884036    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" containerID="cri-o://097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd" gracePeriod=30
Oct 31 03:33:42.169947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.169823    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" podUID="3b629f48-0b1d-470b-9869-08113dce69ec" containerName="payload-processing" containerID="cri-o://9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0" gracePeriod=25
Oct 31 03:33:42.450671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.450331    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" podUID="c6a254da-1023-420e-9556-63959b494ca0" containerName="payload-pre-processing" containerID="cri-o://5e0af7f452ca56229d71301257bcfc75e0c7abaa9a3cbf8c0bdeeebe4d2f9e4a" gracePeriod=25
Oct 31 03:33:42.558386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.558357    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:33:42.644342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.639889    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\") pod \"3b629f48-0b1d-470b-9869-08113dce69ec\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") "
Oct 31 03:33:42.644342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.640107    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\") pod \"3b629f48-0b1d-470b-9869-08113dce69ec\" (UID: \"3b629f48-0b1d-470b-9869-08113dce69ec\") "
Oct 31 03:33:42.648411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.648362    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume" pod "3b629f48-0b1d-470b-9869-08113dce69ec" (UID: "3b629f48-0b1d-470b-9869-08113dce69ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:42.657619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.657564    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m" pod "3b629f48-0b1d-470b-9869-08113dce69ec" (UID: "3b629f48-0b1d-470b-9869-08113dce69ec"). InnerVolumeSpecName "kube-api-access-l5r9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:42.706603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.702372    5831 generic.go:356] "Generic (PLEG): container finished" podID="3b629f48-0b1d-470b-9869-08113dce69ec" containerID="9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0" exitCode=0
Oct 31 03:33:42.706603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.702560    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" event={"ID":"3b629f48-0b1d-470b-9869-08113dce69ec","Type":"ContainerDied","Data":"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"}
Oct 31 03:33:42.706603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.702616    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck" event={"ID":"3b629f48-0b1d-470b-9869-08113dce69ec","Type":"ContainerDied","Data":"af69d4e507520629632133f21e794e59124aa6edd57e56f339f695e16ad1921d"}
Oct 31 03:33:42.706603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.702646    5831 scope.go:122] "RemoveContainer" containerID="9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"
Oct 31 03:33:42.706603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.702802    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"
Oct 31 03:33:42.719762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.708647    5831 generic.go:356] "Generic (PLEG): container finished" podID="c6a254da-1023-420e-9556-63959b494ca0" containerID="5e0af7f452ca56229d71301257bcfc75e0c7abaa9a3cbf8c0bdeeebe4d2f9e4a" exitCode=0
Oct 31 03:33:42.719762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.715416    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" event={"ID":"c6a254da-1023-420e-9556-63959b494ca0","Type":"ContainerDied","Data":"5e0af7f452ca56229d71301257bcfc75e0c7abaa9a3cbf8c0bdeeebe4d2f9e4a"}
Oct 31 03:33:42.728806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.728772    5831 scope.go:122] "RemoveContainer" containerID="9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"
Oct 31 03:33:42.729178 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:42.729153    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0\": container with ID starting with 9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0 not found: ID does not exist" containerID="9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"
Oct 31 03:33:42.729252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.729189    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0"} err="failed to get container status \"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0\": rpc error: code = NotFound desc = could not find container \"9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0\": container with ID starting with 9749b96fafef9745ea444cc57d82d461a5e0ba7646d2c20845feeb681c5a95c0 not found: ID does not exist"
Oct 31 03:33:42.741374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.741347    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:33:42.748559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.748520    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b629f48-0b1d-470b-9869-08113dce69ec-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:42.748559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.748561    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3b629f48-0b1d-470b-9869-08113dce69ec-kube-api-access-l5r9m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:42.850095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.850060    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\" (UniqueName: \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\") pod \"c6a254da-1023-420e-9556-63959b494ca0\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") "
Oct 31 03:33:42.850331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.850311    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\") pod \"c6a254da-1023-420e-9556-63959b494ca0\" (UID: \"c6a254da-1023-420e-9556-63959b494ca0\") "
Oct 31 03:33:42.850684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.850653    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume" pod "c6a254da-1023-420e-9556-63959b494ca0" (UID: "c6a254da-1023-420e-9556-63959b494ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:42.851027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.851006    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a254da-1023-420e-9556-63959b494ca0-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:42.852288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.852271    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf" pod "c6a254da-1023-420e-9556-63959b494ca0" (UID: "c6a254da-1023-420e-9556-63959b494ca0"). InnerVolumeSpecName "kube-api-access-ljbwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:42.952966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.952931    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbwf\" (UniqueName: \"kubernetes.io/projected/c6a254da-1023-420e-9556-63959b494ca0-kube-api-access-ljbwf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:42.964407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.964280    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:33:42.971261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:42.971229    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4fdeb-75d5585fb9-wnhck"]
Oct 31 03:33:42.981273 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:42.981237    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:33:43.575173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.575080    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:33:43.576109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576088    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3b629f48-0b1d-470b-9869-08113dce69ec" containerName="payload-processing"
Oct 31 03:33:43.576217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576112    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b629f48-0b1d-470b-9869-08113dce69ec" containerName="payload-processing"
Oct 31 03:33:43.576217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576139    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c6a254da-1023-420e-9556-63959b494ca0" containerName="payload-pre-processing"
Oct 31 03:33:43.576217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576148    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a254da-1023-420e-9556-63959b494ca0" containerName="payload-pre-processing"
Oct 31 03:33:43.576217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576173    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4b7822da-6223-4e38-9216-cef279a067c9" containerName="payload-pre-processing"
Oct 31 03:33:43.576217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576181    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7822da-6223-4e38-9216-cef279a067c9" containerName="payload-pre-processing"
Oct 31 03:33:43.576384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576221    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" containerName="payload-processing"
Oct 31 03:33:43.576384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576228    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" containerName="payload-processing"
Oct 31 03:33:43.576384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576364    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65f3b0d-d812-4b9a-9c42-78d8cf54e95a" containerName="payload-processing"
Oct 31 03:33:43.576384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576380    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b629f48-0b1d-470b-9869-08113dce69ec" containerName="payload-processing"
Oct 31 03:33:43.576515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576392    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a254da-1023-420e-9556-63959b494ca0" containerName="payload-pre-processing"
Oct 31 03:33:43.576515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.576404    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7822da-6223-4e38-9216-cef279a067c9" containerName="payload-pre-processing"
Oct 31 03:33:43.580844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.580821    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.630738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.630690    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b3a29ec-serving-cert\""
Oct 31 03:33:43.630921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.630758    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:33:43.669051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.668466    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmts\" (UniqueName: \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.669051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.668542    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.669051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.668696    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.669051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.668734    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.676793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.676756    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:33:43.676793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.676763    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 03:33:43.677033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.676878    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:33:43.681206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.681178    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:33:43.725931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.725150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln" event={"ID":"c6a254da-1023-420e-9556-63959b494ca0","Type":"ContainerDied","Data":"57af2f2fccb4a725198d27a212f690866e6ff5851796183ef14635dc6fafab35"}
Oct 31 03:33:43.725931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.725222    5831 scope.go:122] "RemoveContainer" containerID="5e0af7f452ca56229d71301257bcfc75e0c7abaa9a3cbf8c0bdeeebe4d2f9e4a"
Oct 31 03:33:43.725931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.725384    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"
Oct 31 03:33:43.770251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.770218    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.770410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.770260    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.770410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.770379    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmts\" (UniqueName: \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.770562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.770411    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.770562 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:43.770432    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-serving-cert: secret "maas-api-e2e-mt-6b3a29ec-serving-cert" not found
Oct 31 03:33:43.770562 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:43.770551    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls podName:d046121f-f7f2-4d25-a8db-f615c1cf2832 nodeName:}" failed. No retries permitted until 2026-10-31 03:33:44.270514507 +0000 UTC m=+1574.285076193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls") pod "maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" (UID: "d046121f-f7f2-4d25-a8db-f615c1cf2832") : secret "maas-api-e2e-mt-6b3a29ec-serving-cert" not found
Oct 31 03:33:43.771232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.771207    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.773048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.773025    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.901098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.901014    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:33:43.912554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.908821    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmts\" (UniqueName: \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:43.919557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:43.919439    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4fdeb-64fb9df46c-msmln"]
Oct 31 03:33:44.171113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.171033    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b629f48-0b1d-470b-9869-08113dce69ec" path="/var/lib/kubelet/pods/3b629f48-0b1d-470b-9869-08113dce69ec/volumes"
Oct 31 03:33:44.171541 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.171462    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a254da-1023-420e-9556-63959b494ca0" path="/var/lib/kubelet/pods/c6a254da-1023-420e-9556-63959b494ca0/volumes"
Oct 31 03:33:44.292830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.292796    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:44.295691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.295666    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") pod \"maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:44.499462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.499419    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:44.886769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.886732    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:33:44.888728 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:44.888690    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd046121f_f7f2_4d25_a8db_f615c1cf2832.slice/crio-fb349a51873bb98c1a9155bc96c86b59dfddeada1de393931892ddf5a8567d82 WatchSource:0}: Error finding container fb349a51873bb98c1a9155bc96c86b59dfddeada1de393931892ddf5a8567d82: Status 404 returned error can't find the container with id fb349a51873bb98c1a9155bc96c86b59dfddeada1de393931892ddf5a8567d82
Oct 31 03:33:44.891618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:44.891599    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:33:45.746119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:45.746085    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" event={"ID":"d046121f-f7f2-4d25-a8db-f615c1cf2832","Type":"ContainerStarted","Data":"fb349a51873bb98c1a9155bc96c86b59dfddeada1de393931892ddf5a8567d82"}
Oct 31 03:33:45.974058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:45.974022    5831 patch_prober.go:29] interesting pod/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:33:45.974221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:45.974094    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:33:47.674749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.673593    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:33:47.790593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.778739    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:33:47.828060 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.827973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.828334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.828313    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.828474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.828459    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.828673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.828655    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.828871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.828856    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.829095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.829077    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.829263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.829247    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.829404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.829389    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.829567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.829552    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\") pod \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\" (UID: \"6e453fae-fa07-4c20-ae2d-71b1abd1af98\") "
Oct 31 03:33:47.846994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.846940    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:47.860686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.860644    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:47.863646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.860926    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:47.863646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.861095    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:47.863646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.861600    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "kube-api-access-hpksb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:47.864174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.864151    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:47.865820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.865798    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:47.866007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.865763    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:47.869769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.868106    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerID="097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd" exitCode=0
Oct 31 03:33:47.869769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.868270    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerDied","Data":"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"}
Oct 31 03:33:47.869769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.868312    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx" event={"ID":"6e453fae-fa07-4c20-ae2d-71b1abd1af98","Type":"ContainerDied","Data":"47e3a745e4ae05ad6d4bf4a3a2b5a0413e25aa75cfc7ccfe3396e2e9ce03feae"}
Oct 31 03:33:47.869769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.868333    5831 scope.go:122] "RemoveContainer" containerID="097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"
Oct 31 03:33:47.877187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.877167    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:33:47.878361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.878322    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.888739    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token" pod "6e453fae-fa07-4c20-ae2d-71b1abd1af98" (UID: "6e453fae-fa07-4c20-ae2d-71b1abd1af98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.899663    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900628    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900645    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900714    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900722    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900878    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.900899    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" containerName="istio-proxy"
Oct 31 03:33:47.908758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.904717    5831 scope.go:122] "RemoveContainer" containerID="695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"
Oct 31 03:33:47.914424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.913943    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-87af9c\""
Oct 31 03:33:47.919046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.918654    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:33:47.919046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.918803    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:47.930957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.930937    5831 scope.go:122] "RemoveContainer" containerID="097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931669    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgdf\" (UniqueName: \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931750    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931863    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931879    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931891    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931905    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931918    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931932    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931945    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpksb\" (UniqueName: \"kubernetes.io/projected/6e453fae-fa07-4c20-ae2d-71b1abd1af98-kube-api-access-hpksb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931958    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e453fae-fa07-4c20-ae2d-71b1abd1af98-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.931970    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e453fae-fa07-4c20-ae2d-71b1abd1af98-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:47.932011    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd\": container with ID starting with 097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd not found: ID does not exist" containerID="097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.932055    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd"} err="failed to get container status \"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd\": rpc error: code = NotFound desc = could not find container \"097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd\": container with ID starting with 097d845935f62b3f1b3613813da1b6c6ee3ff1601f154fb7bd1b143601d7bedd not found: ID does not exist"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.932082    5831 scope.go:122] "RemoveContainer" containerID="695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:33:47.932406    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6\": container with ID starting with 695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6 not found: ID does not exist" containerID="695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"
Oct 31 03:33:47.933556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.932442    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6"} err="failed to get container status \"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6\": rpc error: code = NotFound desc = could not find container \"695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6\": container with ID starting with 695b65a1b087d39f058309f2b80be713d0723931f910fced4dafa087289e99f6 not found: ID does not exist"
Oct 31 03:33:47.956489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:47.956461    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-87af9c-dockercfg-ztbcz\""
Oct 31 03:33:48.033607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.033575    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvh7\" (UniqueName: \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.033788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.033734    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.033885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.033811    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgdf\" (UniqueName: \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.033885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.033872    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.034621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.034603    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.086857 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.086772    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgdf\" (UniqueName: \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\") pod \"payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.135567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.135519    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.135734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.135663    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvh7\" (UniqueName: \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.136391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.136366    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.189800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.189771    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvh7\" (UniqueName: \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\") pod \"payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") " pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.224658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.224336    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.240960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.240928    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.611122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.611034    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:33:48.650168 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:48.650143    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd379b516_a028_49b2_8e4b_e9637de37eea.slice/crio-fa3d843ade9292528edf8c46abec686380f8a6e0c56b1f0a8cc05870ba20cfbf WatchSource:0}: Error finding container fa3d843ade9292528edf8c46abec686380f8a6e0c56b1f0a8cc05870ba20cfbf: Status 404 returned error can't find the container with id fa3d843ade9292528edf8c46abec686380f8a6e0c56b1f0a8cc05870ba20cfbf
Oct 31 03:33:48.660742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.660717    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:33:48.664669 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:33:48.664644    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef5f7f0_03ff_4e6e_b451_39b3b4ebb246.slice/crio-01e61ba21b9946f321b26bbd03f2d4d0efa3b980f71b51b38b564297add3f198 WatchSource:0}: Error finding container 01e61ba21b9946f321b26bbd03f2d4d0efa3b980f71b51b38b564297add3f198: Status 404 returned error can't find the container with id 01e61ba21b9946f321b26bbd03f2d4d0efa3b980f71b51b38b564297add3f198
Oct 31 03:33:48.877628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.877538    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" event={"ID":"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246","Type":"ContainerStarted","Data":"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"}
Oct 31 03:33:48.877628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.877605    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:33:48.878148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.877629    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" event={"ID":"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246","Type":"ContainerStarted","Data":"01e61ba21b9946f321b26bbd03f2d4d0efa3b980f71b51b38b564297add3f198"}
Oct 31 03:33:48.880465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.880435    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" event={"ID":"d379b516-a028-49b2-8e4b-e9637de37eea","Type":"ContainerStarted","Data":"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"}
Oct 31 03:33:48.880644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.880495    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:48.880644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.880610    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" event={"ID":"d379b516-a028-49b2-8e4b-e9637de37eea","Type":"ContainerStarted","Data":"fa3d843ade9292528edf8c46abec686380f8a6e0c56b1f0a8cc05870ba20cfbf"}
Oct 31 03:33:48.881979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.881946    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"
Oct 31 03:33:48.983371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:48.983304    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" podStartSLOduration=1.983285815 podStartE2EDuration="1.983285815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:48.982480294 +0000 UTC m=+1578.997042002" watchObservedRunningTime="2026-10-31 03:33:48.983285815 +0000 UTC m=+1578.997847524"
Oct 31 03:33:49.045873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:49.045831    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"]
Oct 31 03:33:49.049570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:49.049544    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39186f42-8b882"]
Oct 31 03:33:49.091201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:49.091133    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" podStartSLOduration=2.091115094 podStartE2EDuration="2.091115094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:49.089175186 +0000 UTC m=+1579.103736891" watchObservedRunningTime="2026-10-31 03:33:49.091115094 +0000 UTC m=+1579.105676802"
Oct 31 03:33:49.200463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:49.200417    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:33:49.207116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:49.207087    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-070d07-openshift-default-65f7d87779-6qltx"]
Oct 31 03:33:50.138288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:50.138201    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235b8811-eb3c-4d47-bce6-9095e52520e0" path="/var/lib/kubelet/pods/235b8811-eb3c-4d47-bce6-9095e52520e0/volumes"
Oct 31 03:33:50.138929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:50.138908    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e453fae-fa07-4c20-ae2d-71b1abd1af98" path="/var/lib/kubelet/pods/6e453fae-fa07-4c20-ae2d-71b1abd1af98/volumes"
Oct 31 03:33:50.903202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:50.903163    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" event={"ID":"d046121f-f7f2-4d25-a8db-f615c1cf2832","Type":"ContainerStarted","Data":"70f2247aa9386cc42cd598c3c07f6441c14f806a28e6026cb30c4b8823a28975"}
Oct 31 03:33:50.903635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:50.903613    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:51.014100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:51.014044    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" podStartSLOduration=3.076135395 podStartE2EDuration="8.01402853s" totalImagesPullingTime="4.937893135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:51.010782487 +0000 UTC m=+1581.025344195" watchObservedRunningTime="2026-10-31 03:33:51.01402853 +0000 UTC m=+1581.028590238"
Oct 31 03:33:56.916613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:56.916583    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:33:59.896126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:59.896099    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:33:59.896586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:33:59.896163    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:34:00.042784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:00.042755    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"]
Oct 31 03:34:00.051459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:00.051427    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7def1a8-w6s6c"]
Oct 31 03:34:00.142607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:00.142572    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd298c3-0d0b-48e3-8d28-f0033803ce09" path="/var/lib/kubelet/pods/dbd298c3-0d0b-48e3-8d28-f0033803ce09/volumes"
Oct 31 03:34:07.048047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:07.048013    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"]
Oct 31 03:34:07.092011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:07.091977    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbd0957-h4f6s"]
Oct 31 03:34:08.136744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:08.136712    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c314b4c-cf85-4376-98aa-e19043b47dd3" path="/var/lib/kubelet/pods/0c314b4c-cf85-4376-98aa-e19043b47dd3/volumes"
Oct 31 03:34:11.057892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:11.057857    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:34:11.058395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:11.058138    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" containerID="cri-o://cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6" gracePeriod=30
Oct 31 03:34:11.404604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:11.401717    5831 patch_prober.go:29] interesting pod/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:34:11.404604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:11.401822    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:34:12.978545 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:12.978499    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:34:15.463149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:15.463110    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:34:16.446734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.446705    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.543899    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544007    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544088    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544107    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544125    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544172    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544223    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.544562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.544270    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\") pod \"42c152d0-19f3-4c1c-b265-4ec3158f4022\" (UID: \"42c152d0-19f3-4c1c-b265-4ec3158f4022\") "
Oct 31 03:34:16.545813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.545768    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:16.545942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.545914    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:16.546076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.546057    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:16.546806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.546773    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:16.547041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.547016    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:16.552602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.550634    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:16.562829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.562772    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "kube-api-access-tncqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:16.563959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.563900    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:16.573578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.573071    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token" pod "42c152d0-19f3-4c1c-b265-4ec3158f4022" (UID: "42c152d0-19f3-4c1c-b265-4ec3158f4022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:16.834291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834252    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834557    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834662    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834754    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834771    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncqx\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-kube-api-access-tncqx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834786    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834801    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c152d0-19f3-4c1c-b265-4ec3158f4022-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834816    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c152d0-19f3-4c1c-b265-4ec3158f4022-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:16.834849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:16.834830    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c152d0-19f3-4c1c-b265-4ec3158f4022-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:17.215636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.215481    5831 generic.go:356] "Generic (PLEG): container finished" podID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerID="cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6" exitCode=0
Oct 31 03:34:17.215636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.215562    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"
Oct 31 03:34:17.215636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.215604    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerDied","Data":"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"}
Oct 31 03:34:17.215909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.215652    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt" event={"ID":"42c152d0-19f3-4c1c-b265-4ec3158f4022","Type":"ContainerDied","Data":"c966acf7352473ce3b9aaa479f969caa3589f69cc27a1d4782a7e8276e27467e"}
Oct 31 03:34:17.215909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.215686    5831 scope.go:122] "RemoveContainer" containerID="cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"
Oct 31 03:34:17.229287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.229253    5831 scope.go:122] "RemoveContainer" containerID="9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"
Oct 31 03:34:17.240205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.240185    5831 scope.go:122] "RemoveContainer" containerID="cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"
Oct 31 03:34:17.240616 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:17.240442    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6\": container with ID starting with cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6 not found: ID does not exist" containerID="cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"
Oct 31 03:34:17.240727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.240625    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6"} err="failed to get container status \"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6\": rpc error: code = NotFound desc = could not find container \"cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6\": container with ID starting with cb3f735631c50b00bcffd676166e4bd781098076bdfcb117800c626bbe604ee6 not found: ID does not exist"
Oct 31 03:34:17.240727 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.240644    5831 scope.go:122] "RemoveContainer" containerID="9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"
Oct 31 03:34:17.241143 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:17.241121    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864\": container with ID starting with 9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864 not found: ID does not exist" containerID="9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"
Oct 31 03:34:17.241233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.241150    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864"} err="failed to get container status \"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864\": rpc error: code = NotFound desc = could not find container \"9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864\": container with ID starting with 9081aff26e24ff0e2f547e51d31a844b599b1f93cad88b4e8477b4e4265a7864 not found: ID does not exist"
Oct 31 03:34:17.368430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.368400    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:34:17.424969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.424549    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:34:17.435331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.435297    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4fdeb-openshift-default-78966fb9c5-wcndt"]
Oct 31 03:34:17.514197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:17.514108    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:34:18.137751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:18.137547    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" path="/var/lib/kubelet/pods/42c152d0-19f3-4c1c-b265-4ec3158f4022/volumes"
Oct 31 03:34:22.369204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.369138    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" podUID="23f44818-267b-4a30-9ed6-dd850f7e58ae" containerName="payload-processing" containerID="cri-o://c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d" gracePeriod=25
Oct 31 03:34:22.515412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.515367    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" podUID="d836abeb-2c33-4b93-90f9-db595d9003f7" containerName="payload-pre-processing" containerID="cri-o://7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6" gracePeriod=25
Oct 31 03:34:22.659200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.659174    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:34:22.749077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.749044    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\") pod \"23f44818-267b-4a30-9ed6-dd850f7e58ae\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") "
Oct 31 03:34:22.749279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.749107    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\" (UniqueName: \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\") pod \"23f44818-267b-4a30-9ed6-dd850f7e58ae\" (UID: \"23f44818-267b-4a30-9ed6-dd850f7e58ae\") "
Oct 31 03:34:22.749470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.749445    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume" pod "23f44818-267b-4a30-9ed6-dd850f7e58ae" (UID: "23f44818-267b-4a30-9ed6-dd850f7e58ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:22.750018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.749994    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f44818-267b-4a30-9ed6-dd850f7e58ae-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:22.752045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.752017    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs" pod "23f44818-267b-4a30-9ed6-dd850f7e58ae" (UID: "23f44818-267b-4a30-9ed6-dd850f7e58ae"). InnerVolumeSpecName "kube-api-access-p5lcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:22.815427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.815393    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:34:22.850823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.850792    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\") pod \"d836abeb-2c33-4b93-90f9-db595d9003f7\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") "
Oct 31 03:34:22.851036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.851023    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\" (UniqueName: \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\") pod \"d836abeb-2c33-4b93-90f9-db595d9003f7\" (UID: \"d836abeb-2c33-4b93-90f9-db595d9003f7\") "
Oct 31 03:34:22.851186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.851163    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume" pod "d836abeb-2c33-4b93-90f9-db595d9003f7" (UID: "d836abeb-2c33-4b93-90f9-db595d9003f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:22.851356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.851341    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lcs\" (UniqueName: \"kubernetes.io/projected/23f44818-267b-4a30-9ed6-dd850f7e58ae-kube-api-access-p5lcs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:22.851395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.851358    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d836abeb-2c33-4b93-90f9-db595d9003f7-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:22.853349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.853327    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv" pod "d836abeb-2c33-4b93-90f9-db595d9003f7" (UID: "d836abeb-2c33-4b93-90f9-db595d9003f7"). InnerVolumeSpecName "kube-api-access-rg4kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:22.952687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:22.952646    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4kv\" (UniqueName: \"kubernetes.io/projected/d836abeb-2c33-4b93-90f9-db595d9003f7-kube-api-access-rg4kv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:23.282832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.282660    5831 generic.go:356] "Generic (PLEG): container finished" podID="d836abeb-2c33-4b93-90f9-db595d9003f7" containerID="7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6" exitCode=0
Oct 31 03:34:23.282832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.282699    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"
Oct 31 03:34:23.282832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.282739    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" event={"ID":"d836abeb-2c33-4b93-90f9-db595d9003f7","Type":"ContainerDied","Data":"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"}
Oct 31 03:34:23.282832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.282798    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn" event={"ID":"d836abeb-2c33-4b93-90f9-db595d9003f7","Type":"ContainerDied","Data":"f9f411302a7d1e55e0e1fb1665707e04d26baeefdda4cfd560ffbe62fdef509a"}
Oct 31 03:34:23.283225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.282857    5831 scope.go:122] "RemoveContainer" containerID="7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"
Oct 31 03:34:23.286187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.286163    5831 generic.go:356] "Generic (PLEG): container finished" podID="23f44818-267b-4a30-9ed6-dd850f7e58ae" containerID="c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d" exitCode=0
Oct 31 03:34:23.286309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.286226    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"
Oct 31 03:34:23.286309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.286280    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" event={"ID":"23f44818-267b-4a30-9ed6-dd850f7e58ae","Type":"ContainerDied","Data":"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"}
Oct 31 03:34:23.286422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.286335    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn" event={"ID":"23f44818-267b-4a30-9ed6-dd850f7e58ae","Type":"ContainerDied","Data":"01f6f664d9b42bf264dbdcd82f0ccb49598cbb4ad704be6060d177db8e727a81"}
Oct 31 03:34:23.302825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.302808    5831 scope.go:122] "RemoveContainer" containerID="7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"
Oct 31 03:34:23.303167 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:23.303145    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6\": container with ID starting with 7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6 not found: ID does not exist" containerID="7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"
Oct 31 03:34:23.303281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.303173    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6"} err="failed to get container status \"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6\": rpc error: code = NotFound desc = could not find container \"7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6\": container with ID starting with 7e910cce281401f1ded7b2e7096dbcae7c2511e1bef65e072b7507369767d9e6 not found: ID does not exist"
Oct 31 03:34:23.303281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.303191    5831 scope.go:122] "RemoveContainer" containerID="c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"
Oct 31 03:34:23.313724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.313704    5831 scope.go:122] "RemoveContainer" containerID="c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"
Oct 31 03:34:23.314013 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:23.313996    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d\": container with ID starting with c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d not found: ID does not exist" containerID="c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"
Oct 31 03:34:23.314097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.314021    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d"} err="failed to get container status \"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d\": rpc error: code = NotFound desc = could not find container \"c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d\": container with ID starting with c09dbbcf34776f630f0c11ef5280d9e4c718e4a320928d13468d07cf134ee14d not found: ID does not exist"
Oct 31 03:34:23.436707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.436676    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:34:23.483824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.483787    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2260d8-85d8f74545-kmhzn"]
Oct 31 03:34:23.541265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.541173    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:34:23.544761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:23.544737    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2260d8-5594b76647-zrzgn"]
Oct 31 03:34:24.135940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.135905    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f44818-267b-4a30-9ed6-dd850f7e58ae" path="/var/lib/kubelet/pods/23f44818-267b-4a30-9ed6-dd850f7e58ae/volumes"
Oct 31 03:34:24.136343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.136330    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d836abeb-2c33-4b93-90f9-db595d9003f7" path="/var/lib/kubelet/pods/d836abeb-2c33-4b93-90f9-db595d9003f7/volumes"
Oct 31 03:34:24.685881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.685839    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:34:24.686802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686782    5831 cpu_manager.go:414] "containerMap: removing container" podUID="23f44818-267b-4a30-9ed6-dd850f7e58ae" containerName="payload-processing"
Oct 31 03:34:24.686886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686804    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f44818-267b-4a30-9ed6-dd850f7e58ae" containerName="payload-processing"
Oct 31 03:34:24.686886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686837    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d836abeb-2c33-4b93-90f9-db595d9003f7" containerName="payload-pre-processing"
Oct 31 03:34:24.686886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686845    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d836abeb-2c33-4b93-90f9-db595d9003f7" containerName="payload-pre-processing"
Oct 31 03:34:24.687040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686892    5831 cpu_manager.go:414] "containerMap: removing container" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:24.687040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686900    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:24.687040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686931    5831 cpu_manager.go:414] "containerMap: removing container" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:24.687040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.686938    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:24.687196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.687072    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:24.687196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.687088    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f44818-267b-4a30-9ed6-dd850f7e58ae" containerName="payload-processing"
Oct 31 03:34:24.687196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.687101    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d836abeb-2c33-4b93-90f9-db595d9003f7" containerName="payload-pre-processing"
Oct 31 03:34:24.692957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.692935    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.786456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.781747    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.910923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.910887    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.915983    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916261    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916444    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916497    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wlcd\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916556    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916614    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.916666    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:24.916851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.865586    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:34:24.916935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:24.901928    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5d3f8a-openshift-default-dockercfg-v7k67\""
Oct 31 03:34:25.060498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060401    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wlcd\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.060498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.060820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060519    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.060820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060588    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.060820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060739    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.060820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060777    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.061054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060842    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.061054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060871    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.061054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.060961    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.069580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.061456    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.069580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.062107    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.069580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.065155    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.076665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.074718    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.076665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.074902    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.088574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.080827    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.152553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.148471    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.174565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.160845    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.180560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.177291    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wlcd\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\") pod \"e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") " pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.311392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.311303    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:25.739880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.737316    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:34:25.742980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.742935    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:25.743072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.743022    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:25.743072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:25.743049    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:26.328210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:26.328169    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerStarted","Data":"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"}
Oct 31 03:34:26.328419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:26.328270    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerStarted","Data":"37302dcf5157cab23c881993026a527b8c32f261c865cc735d1b7710b8946bfc"}
Oct 31 03:34:26.437747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:26.437679    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podStartSLOduration=2.437664473 podStartE2EDuration="2.437664473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:26.435520724 +0000 UTC m=+1616.450082432" watchObservedRunningTime="2026-10-31 03:34:26.437664473 +0000 UTC m=+1616.452226181"
Oct 31 03:34:27.312450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:27.312413    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:27.318587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:27.318557    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:27.339144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:27.339112    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:27.340234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:27.340210    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:34:29.425627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.410321    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:34:29.425627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.411695    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c152d0-19f3-4c1c-b265-4ec3158f4022" containerName="istio-proxy"
Oct 31 03:34:29.431174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.431131    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.483883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.483047    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e1aad0-openshift-default-dockercfg-tlhkf\""
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617232    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617307    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6n8f\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617349    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617393    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617477    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617615    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617674    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617746    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.620556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.617876    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.753608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.748387    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:34:29.753608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.751504    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.856729    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.856819    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.856969    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.857046    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.857077    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6n8f\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.857122    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.857170    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.857251    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.867271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.847147    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.867900    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.868690    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.869155    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.877269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.876629    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.901201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.901168    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.946586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.945133    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6n8f\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:29.953409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:29.952736    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:30.089488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:30.089404    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:30.520445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:30.520419    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:34:30.521657 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:30.521619    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd421a5c4_f284_4f4b_8904_41837c250ead.slice/crio-dd1da56f56c110630ccbada259bfd109beb040e8c47baa7bb156a88d54c95bb3 WatchSource:0}: Error finding container dd1da56f56c110630ccbada259bfd109beb040e8c47baa7bb156a88d54c95bb3: Status 404 returned error can't find the container with id dd1da56f56c110630ccbada259bfd109beb040e8c47baa7bb156a88d54c95bb3
Oct 31 03:34:30.524483 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:30.524447    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:30.524599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:30.524518    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:30.524599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:30.524573    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:31.374656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:31.374048    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerStarted","Data":"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"}
Oct 31 03:34:31.374656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:31.374110    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerStarted","Data":"dd1da56f56c110630ccbada259bfd109beb040e8c47baa7bb156a88d54c95bb3"}
Oct 31 03:34:31.528561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:31.528465    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podStartSLOduration=2.528448356 podStartE2EDuration="2.528448356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:31.527339543 +0000 UTC m=+1621.541901251" watchObservedRunningTime="2026-10-31 03:34:31.528448356 +0000 UTC m=+1621.543010061"
Oct 31 03:34:32.090341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.090302    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:32.096299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.096275    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:32.384754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.384668    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:32.385959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.385932    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:34:32.519315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.519277    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:34:32.524802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.524780    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.583707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.583572    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-87af9c-serving-cert\""
Oct 31 03:34:32.599642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.599607    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.599848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.599820    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.599951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.599880    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhlr\" (UniqueName: \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.600017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.599957    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.608844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.608812    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:34:32.700904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.700875    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.701089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.700960    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.701089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.700992    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhlr\" (UniqueName: \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.701089 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:32.701034    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-serving-cert: secret "maas-api-e2e-api-b-87af9c-serving-cert" not found
Oct 31 03:34:32.701089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.701050    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.701269 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:32.701118    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls podName:cf88afca-7746-4233-8c4e-98a3b2d0131d nodeName:}" failed. No retries permitted until 2026-10-31 03:34:33.201096171 +0000 UTC m=+1623.215657859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls") pod "maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" (UID: "cf88afca-7746-4233-8c4e-98a3b2d0131d") : secret "maas-api-e2e-api-b-87af9c-serving-cert" not found
Oct 31 03:34:32.701804 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.701780    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.703783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.703763    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:32.756596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:32.756559    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhlr\" (UniqueName: \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:33.032854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.032768    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:34:33.037793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.037773    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.088552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.086765    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5d3f8a-dockercfg-fn9vb\""
Oct 31 03:34:33.088552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.086992    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5d3f8a\""
Oct 31 03:34:33.089818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.089790    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:34:33.164554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.163727    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.164554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.163912    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqpp5\" (UniqueName: \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.281687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.280969    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqpp5\" (UniqueName: \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.281687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.281335    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:33.281687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.281371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.284685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.283271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.355557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.342966    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:34:33.373495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.372111    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.413209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.413170    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqpp5\" (UniqueName: \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.416099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.416066    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") pod \"maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:33.417943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.417915    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:34:33.451670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.451641    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:33.521409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.521364    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.521562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.521519    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5rnr\" (UniqueName: \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.622968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.622867    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5rnr\" (UniqueName: \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.623401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.623090    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.623989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.623966    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.654503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.654468    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:33.680255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.680226    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5rnr\" (UniqueName: \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\") pod \"payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.749133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.749103    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:33.769941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.769911    5831 scope.go:122] "RemoveContainer" containerID="46c27379a8f5b87e289179eeb7c31e989db975ddac5e6040b9045e8d1ff7f15c"
Oct 31 03:34:33.809223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.809200    5831 scope.go:122] "RemoveContainer" containerID="53dae62e6b4699fa295c9dd4910ba9b9ed47a92b7af60814c64bfc5bedbc05da"
Oct 31 03:34:33.868660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:33.857175    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:34:33.868660 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:33.863125    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf88afca_7746_4233_8c4e_98a3b2d0131d.slice/crio-b8b834952406db23cb9e023b8812262a452618bad5f70787fe3aceaa6b6a5bd9 WatchSource:0}: Error finding container b8b834952406db23cb9e023b8812262a452618bad5f70787fe3aceaa6b6a5bd9: Status 404 returned error can't find the container with id b8b834952406db23cb9e023b8812262a452618bad5f70787fe3aceaa6b6a5bd9
Oct 31 03:34:34.047661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.047630    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:34:34.049033 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:34.049005    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc7a8b9_92db_447b_b9e5_11beaf7ea97b.slice/crio-fa2bc48ec4c4670df05a23ba3106f21bfe426c067772ffcb6e2c16d587fa8020 WatchSource:0}: Error finding container fa2bc48ec4c4670df05a23ba3106f21bfe426c067772ffcb6e2c16d587fa8020: Status 404 returned error can't find the container with id fa2bc48ec4c4670df05a23ba3106f21bfe426c067772ffcb6e2c16d587fa8020
Oct 31 03:34:34.149345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.149310    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:34:34.151441 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:34.151414    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b024c5_a49b_4e08_a533_44f0656e8aef.slice/crio-bbc026101a3c87abde9e916c65f9c474948fd7e20f25893cd19c0e84a4809204 WatchSource:0}: Error finding container bbc026101a3c87abde9e916c65f9c474948fd7e20f25893cd19c0e84a4809204: Status 404 returned error can't find the container with id bbc026101a3c87abde9e916c65f9c474948fd7e20f25893cd19c0e84a4809204
Oct 31 03:34:34.448339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.448305    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" event={"ID":"54b024c5-a49b-4e08-a533-44f0656e8aef","Type":"ContainerStarted","Data":"9e0c59ce2ecc2051625166b6e3a78a6cccbb05b2f7f527084fed1941c49f3534"}
Oct 31 03:34:34.448613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.448367    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" event={"ID":"54b024c5-a49b-4e08-a533-44f0656e8aef","Type":"ContainerStarted","Data":"bbc026101a3c87abde9e916c65f9c474948fd7e20f25893cd19c0e84a4809204"}
Oct 31 03:34:34.448613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.448542    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:34.450158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.450119    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" event={"ID":"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b","Type":"ContainerStarted","Data":"fa00d3b342dde22ac36dbe66282c0e892028b6babc3ba3ba0abafad4c49c6979"}
Oct 31 03:34:34.450302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.450166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" event={"ID":"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b","Type":"ContainerStarted","Data":"fa2bc48ec4c4670df05a23ba3106f21bfe426c067772ffcb6e2c16d587fa8020"}
Oct 31 03:34:34.450302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.450196    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:34.451768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.451742    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" event={"ID":"cf88afca-7746-4233-8c4e-98a3b2d0131d","Type":"ContainerStarted","Data":"b8b834952406db23cb9e023b8812262a452618bad5f70787fe3aceaa6b6a5bd9"}
Oct 31 03:34:34.561300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.561244    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" podStartSLOduration=1.5612287230000002 podStartE2EDuration="1.561228723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:34.557178452 +0000 UTC m=+1624.571740160" watchObservedRunningTime="2026-10-31 03:34:34.561228723 +0000 UTC m=+1624.575790430"
Oct 31 03:34:34.666177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:34.666109    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" podStartSLOduration=2.666093108 podStartE2EDuration="2.666093108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:34.6625171 +0000 UTC m=+1624.677078808" watchObservedRunningTime="2026-10-31 03:34:34.666093108 +0000 UTC m=+1624.680654815"
Oct 31 03:34:35.460824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:35.460785    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" event={"ID":"cf88afca-7746-4233-8c4e-98a3b2d0131d","Type":"ContainerStarted","Data":"a07e3c2df9001e1db34f603992983f56daa011c8a5339e22deebb3bd1546cded"}
Oct 31 03:34:35.461009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:35.460846    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:35.585943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:35.585880    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" podStartSLOduration=2.6631321420000003 podStartE2EDuration="3.585858524s" totalImagesPullingTime="922.726382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:35.583658633 +0000 UTC m=+1625.598220341" watchObservedRunningTime="2026-10-31 03:34:35.585858524 +0000 UTC m=+1625.600420237"
Oct 31 03:34:37.527214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.527177    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:34:37.527773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.527428    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy" containerID="cri-o://358c9bd42d1170edc9364a2af12b5f1a48bda91e5111faf4c7b7b740939d5c60" gracePeriod=30
Oct 31 03:34:37.573189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.573038    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:34:37.573392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.573318    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy" containerID="cri-o://1cc98893d83b376e0869c44dde588b653f62484e465b3f4065c5e43216e5b188" gracePeriod=30
Oct 31 03:34:37.577422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.577395    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:34:37.577667 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.577624    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy" containerID="cri-o://2badf9ddb7c0ca47af126fee754bf9a0b4c95b79fad35f84c7b29a05d4589cc4" gracePeriod=30
Oct 31 03:34:37.620715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.620684    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:34:37.620937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:37.620914    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" containerID="cri-o://fd0b977931aebe8ba96ffbc7049600bc10e6f6dd0b2ceec4b516e93c52f6b299" gracePeriod=30
Oct 31 03:34:39.344902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:39.344866    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:34:39.345329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:39.344932    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:34:40.470567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.470051    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:34:40.779839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.769993    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.793809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.793780    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:34:40.803971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.803778    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:34:40.809207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.809181    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:40.810295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.810267    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:34:40.821021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.819120    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e1aad0\""
Oct 31 03:34:40.834084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.834055    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.834297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.834279    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdqf\" (UniqueName: \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.865101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.865069    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e1aad0-dockercfg-mpbsh\""
Oct 31 03:34:40.936129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.936097    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmht\" (UniqueName: \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:40.936317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.936214    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:40.936317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.936257    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.936517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.936494    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdqf\" (UniqueName: \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.936992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.936974    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:40.989796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:40.989759    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdqf\" (UniqueName: \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:41.038058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.037971    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmht\" (UniqueName: \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.038220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.038095    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.038893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.038872    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.050125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.050099    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"]
Oct 31 03:34:41.054144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.054117    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63152d-djddc"]
Oct 31 03:34:41.090966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.090934    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmht\" (UniqueName: \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\") pod \"payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.113786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.113750    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:41.125815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.125586    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.474220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.474189    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:34:41.500584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.500558    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:34:41.502000 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:41.501973    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38261dc_5682_45d8_89a9_2320e5196831.slice/crio-5c1e8412618114cea46f100e928e7f4e686d1499c2ba95fec8d06070d4640680 WatchSource:0}: Error finding container 5c1e8412618114cea46f100e928e7f4e686d1499c2ba95fec8d06070d4640680: Status 404 returned error can't find the container with id 5c1e8412618114cea46f100e928e7f4e686d1499c2ba95fec8d06070d4640680
Oct 31 03:34:41.550059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.550030    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:34:41.559628 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:41.559601    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4210d2d9_981a_4e07_a275_4b5e887d70b4.slice/crio-5057960c4f348c6c8ab95222e03d11694c9f1fcc9274f20d514be94af7e42ba3 WatchSource:0}: Error finding container 5057960c4f348c6c8ab95222e03d11694c9f1fcc9274f20d514be94af7e42ba3: Status 404 returned error can't find the container with id 5057960c4f348c6c8ab95222e03d11694c9f1fcc9274f20d514be94af7e42ba3
Oct 31 03:34:41.806176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.806084    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" event={"ID":"d38261dc-5682-45d8-89a9-2320e5196831","Type":"ContainerStarted","Data":"c440897344225ebb874eff1dbb2a410591f924022f5f7ce3fc7f39349f3cf88e"}
Oct 31 03:34:41.806176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.806151    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:41.806176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.806173    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" event={"ID":"d38261dc-5682-45d8-89a9-2320e5196831","Type":"ContainerStarted","Data":"5c1e8412618114cea46f100e928e7f4e686d1499c2ba95fec8d06070d4640680"}
Oct 31 03:34:41.807850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.807824    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" event={"ID":"4210d2d9-981a-4e07-a275-4b5e887d70b4","Type":"ContainerStarted","Data":"0b938a2ae2fe88a5a7da72c9e3766dd2b9b631c209b6373fb647fed037cef269"}
Oct 31 03:34:41.808004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.807868    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" event={"ID":"4210d2d9-981a-4e07-a275-4b5e887d70b4","Type":"ContainerStarted","Data":"5057960c4f348c6c8ab95222e03d11694c9f1fcc9274f20d514be94af7e42ba3"}
Oct 31 03:34:41.808004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.807890    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:41.916040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:41.915992    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" podStartSLOduration=1.915978475 podStartE2EDuration="1.915978475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:41.911904276 +0000 UTC m=+1631.926465985" watchObservedRunningTime="2026-10-31 03:34:41.915978475 +0000 UTC m=+1631.930540182"
Oct 31 03:34:42.021188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.021124    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" podStartSLOduration=2.021104423 podStartE2EDuration="2.021104423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:42.016910626 +0000 UTC m=+1632.031472335" watchObservedRunningTime="2026-10-31 03:34:42.021104423 +0000 UTC m=+1632.035666134"
Oct 31 03:34:42.204555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.202754    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aac3aa0-bfd6-4b75-b2fa-52a302bc6564" path="/var/lib/kubelet/pods/3aac3aa0-bfd6-4b75-b2fa-52a302bc6564/volumes"
Oct 31 03:34:42.828606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.828163    5831 generic.go:356] "Generic (PLEG): container finished" podID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerID="fd0b977931aebe8ba96ffbc7049600bc10e6f6dd0b2ceec4b516e93c52f6b299" exitCode=0
Oct 31 03:34:42.828606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.828431    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" event={"ID":"5b008e5f-6f87-4bba-80f1-a21818782fa9","Type":"ContainerDied","Data":"fd0b977931aebe8ba96ffbc7049600bc10e6f6dd0b2ceec4b516e93c52f6b299"}
Oct 31 03:34:42.832898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.832862    5831 generic.go:356] "Generic (PLEG): container finished" podID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerID="1cc98893d83b376e0869c44dde588b653f62484e465b3f4065c5e43216e5b188" exitCode=0
Oct 31 03:34:42.833019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.832934    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" event={"ID":"91d64a96-6a85-490a-9f7e-377d9ab3ca18","Type":"ContainerDied","Data":"1cc98893d83b376e0869c44dde588b653f62484e465b3f4065c5e43216e5b188"}
Oct 31 03:34:42.834696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.834671    5831 generic.go:356] "Generic (PLEG): container finished" podID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerID="2badf9ddb7c0ca47af126fee754bf9a0b4c95b79fad35f84c7b29a05d4589cc4" exitCode=0
Oct 31 03:34:42.834805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.834788    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" event={"ID":"5906ea5c-3b72-4661-b0a0-663d0fa093bb","Type":"ContainerDied","Data":"2badf9ddb7c0ca47af126fee754bf9a0b4c95b79fad35f84c7b29a05d4589cc4"}
Oct 31 03:34:42.836348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.836263    5831 generic.go:356] "Generic (PLEG): container finished" podID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerID="358c9bd42d1170edc9364a2af12b5f1a48bda91e5111faf4c7b7b740939d5c60" exitCode=0
Oct 31 03:34:42.839576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.837142    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" event={"ID":"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1","Type":"ContainerDied","Data":"358c9bd42d1170edc9364a2af12b5f1a48bda91e5111faf4c7b7b740939d5c60"}
Oct 31 03:34:42.886954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.886637    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987071    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987115    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987190    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987308    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987382    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987444    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987467    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987512    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.987710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987613    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\") pod \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\" (UID: \"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1\") "
Oct 31 03:34:42.988362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.987861    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.988362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988282    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:42.988474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988383    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.988563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988484    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.994612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988819    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.994612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988844    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.994612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.988859    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.994612 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:42.989294    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:34:42.997163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.997118    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "kube-api-access-ds6hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:42.997694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.997646    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:42.998021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.997950    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.999512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:42.999485    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo" pod "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" (UID: "fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:43.001838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.001818    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:34:43.043251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.043214    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:34:43.046943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.046920    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090295    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090389    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090421    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090461    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090622    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090696    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090818    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.090865    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\") pod \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\" (UID: \"91d64a96-6a85-490a-9f7e-377d9ab3ca18\") "
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091323    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091343    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091360    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091375    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091394    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091410    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds6hj\" (UniqueName: \"kubernetes.io/projected/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1-kube-api-access-ds6hj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.091802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.091699    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.092791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.092108    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.092791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.092161    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:43.092791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.092511    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.092791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.092771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.095229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.095193    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:43.095399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.095377    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.095617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.095596    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.096214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.096186    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f" pod "91d64a96-6a85-490a-9f7e-377d9ab3ca18" (UID: "91d64a96-6a85-490a-9f7e-377d9ab3ca18"). InnerVolumeSpecName "kube-api-access-6df8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207024    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207131    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207183    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207254    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207295    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207348    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207383    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207429    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207544    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207589    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207656    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207715    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207777    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207839    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207869    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207906    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\") pod \"5b008e5f-6f87-4bba-80f1-a21818782fa9\" (UID: \"5b008e5f-6f87-4bba-80f1-a21818782fa9\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.207965    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\") pod \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\" (UID: \"5906ea5c-3b72-4661-b0a0-663d0fa093bb\") "
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208575    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208596    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208609    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6df8f\" (UniqueName: \"kubernetes.io/projected/91d64a96-6a85-490a-9f7e-377d9ab3ca18-kube-api-access-6df8f\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208625    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208638    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208652    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208666    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208681    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.223591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.208696    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d64a96-6a85-490a-9f7e-377d9ab3ca18-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.246279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.239761    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.258553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.250953    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:43.258553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.254660    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.276870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.276576    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.288643    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.288832    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.288930    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "kube-api-access-bdq2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.289599    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333502    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333558    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333576    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333590    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333605    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333619    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdq2m\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-kube-api-access-bdq2m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333634    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.333647    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.334573    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.337460    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.337690    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.348004    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.388617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.348456    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:43.403023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.402967    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.432376    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.433185    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435084    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b008e5f-6f87-4bba-80f1-a21818782fa9-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435112    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435127    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435140    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435153    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435168    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435181    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b008e5f-6f87-4bba-80f1-a21818782fa9-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.435484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.435194    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b008e5f-6f87-4bba-80f1-a21818782fa9-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.444595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.442712    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo" pod "5906ea5c-3b72-4661-b0a0-663d0fa093bb" (UID: "5906ea5c-3b72-4661-b0a0-663d0fa093bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:43.444595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.442867    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb" pod "5b008e5f-6f87-4bba-80f1-a21818782fa9" (UID: "5b008e5f-6f87-4bba-80f1-a21818782fa9"). InnerVolumeSpecName "kube-api-access-f96kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:43.536264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.536232    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5906ea5c-3b72-4661-b0a0-663d0fa093bb-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.536264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.536263    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96kb\" (UniqueName: \"kubernetes.io/projected/5b008e5f-6f87-4bba-80f1-a21818782fa9-kube-api-access-f96kb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.845227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.845195    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7" event={"ID":"91d64a96-6a85-490a-9f7e-377d9ab3ca18","Type":"ContainerDied","Data":"96d0399937c8e710f73df4eb00b9d4dc56c49c3d528ee564509f3aa74365472b"}
Oct 31 03:34:43.845227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.845211    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"
Oct 31 03:34:43.845806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.845280    5831 scope.go:122] "RemoveContainer" containerID="1cc98893d83b376e0869c44dde588b653f62484e465b3f4065c5e43216e5b188"
Oct 31 03:34:43.846919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.846894    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7" event={"ID":"5906ea5c-3b72-4661-b0a0-663d0fa093bb","Type":"ContainerDied","Data":"77de1f685b7a4a11296e9785904b0b78c77f4c2a519982be44f9361ef4afdea5"}
Oct 31 03:34:43.847056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.846905    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"
Oct 31 03:34:43.848696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.848654    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx" event={"ID":"fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1","Type":"ContainerDied","Data":"49abf93037d755d6e9d354d9b1b5645d4d0d23d2b10c154e365f6432e3ee34e5"}
Oct 31 03:34:43.848825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.848675    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"
Oct 31 03:34:43.850579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.850559    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"
Oct 31 03:34:43.850704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.850572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9" event={"ID":"5b008e5f-6f87-4bba-80f1-a21818782fa9","Type":"ContainerDied","Data":"8b667100bbb0437321fbbabd934b24b4a962ff036b9f96510c16b0e5c039935f"}
Oct 31 03:34:43.863813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.863741    5831 scope.go:122] "RemoveContainer" containerID="2badf9ddb7c0ca47af126fee754bf9a0b4c95b79fad35f84c7b29a05d4589cc4"
Oct 31 03:34:43.875448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.875428    5831 scope.go:122] "RemoveContainer" containerID="358c9bd42d1170edc9364a2af12b5f1a48bda91e5111faf4c7b7b740939d5c60"
Oct 31 03:34:43.886187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:43.886165    5831 scope.go:122] "RemoveContainer" containerID="fd0b977931aebe8ba96ffbc7049600bc10e6f6dd0b2ceec4b516e93c52f6b299"
Oct 31 03:34:44.001241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.000630    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:34:44.005373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.005353    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-d5dfx"]
Oct 31 03:34:44.109271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.109238    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:34:44.114960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.114930    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-tnqm7"]
Oct 31 03:34:44.138070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.138022    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" path="/var/lib/kubelet/pods/5906ea5c-3b72-4661-b0a0-663d0fa093bb/volumes"
Oct 31 03:34:44.140111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.138913    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" path="/var/lib/kubelet/pods/fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1/volumes"
Oct 31 03:34:44.224662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.224593    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:34:44.229808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.229777    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zzfd9"]
Oct 31 03:34:44.326444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.326353    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:34:44.331421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:44.331395    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-hrkn7"]
Oct 31 03:34:45.464386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:45.464358    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:34:45.464822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:45.464423    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:34:46.146652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:46.146615    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" path="/var/lib/kubelet/pods/5b008e5f-6f87-4bba-80f1-a21818782fa9/volumes"
Oct 31 03:34:46.147403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:46.147376    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" path="/var/lib/kubelet/pods/91d64a96-6a85-490a-9f7e-377d9ab3ca18/volumes"
Oct 31 03:34:48.363403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:48.363334    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:34:49.074670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.074613    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:34:49.075929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.075901    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy"
Oct 31 03:34:49.076120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076106    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy"
Oct 31 03:34:49.076226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076216    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy"
Oct 31 03:34:49.076299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076289    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy"
Oct 31 03:34:49.076376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076367    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy"
Oct 31 03:34:49.076450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076441    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy"
Oct 31 03:34:49.076594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076522    5831 cpu_manager.go:414] "containerMap: removing container" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy"
Oct 31 03:34:49.076687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.076676    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy"
Oct 31 03:34:49.077065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.077048    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d64a96-6a85-490a-9f7e-377d9ab3ca18" containerName="istio-proxy"
Oct 31 03:34:49.089406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.089384    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdca36f-fcac-41b5-ab0b-fe3f0751f4d1" containerName="istio-proxy"
Oct 31 03:34:49.089595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.089581    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5906ea5c-3b72-4661-b0a0-663d0fa093bb" containerName="istio-proxy"
Oct 31 03:34:49.089704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.089694    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b008e5f-6f87-4bba-80f1-a21818782fa9" containerName="istio-proxy"
Oct 31 03:34:49.096381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.096357    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.197674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.197638    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:34:49.245471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.245434    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.245731 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.245712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.245971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.245936    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppdg\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246080    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246112    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246147    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246203    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246235    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.246333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.246262    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.347916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.347821    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.348642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.348617    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.349233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.349213    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.349767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.349750    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.350624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.350601    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.351061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.351044    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.351316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.351301    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.351463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.351449    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.351673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.351655    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppdg\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.353031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.352372    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.353169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.348520    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.353284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.349655    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.353395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.350514    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.353497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.350999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.357059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.357036    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.367002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.366973    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.406362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.406294    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppdg\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.411272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.411190    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.467083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.467043    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:49.510652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.510597    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7199-8f517a"]
Oct 31 03:34:49.516506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.516482    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:49.545290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.545261    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7199-8f517a"]
Oct 31 03:34:49.566284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.566237    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4jjr\""
Oct 31 03:34:49.659606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.659478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg25d\" (UniqueName: \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\") pod \"test-curl-7199-8f517a\" (UID: \"9291ca0b-465f-4d80-95d1-774ee938deb1\") " pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:49.773925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.773850    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg25d\" (UniqueName: \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\") pod \"test-curl-7199-8f517a\" (UID: \"9291ca0b-465f-4d80-95d1-774ee938deb1\") " pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:49.832422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.832392    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg25d\" (UniqueName: \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\") pod \"test-curl-7199-8f517a\" (UID: \"9291ca0b-465f-4d80-95d1-774ee938deb1\") " pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:49.835457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.835273    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:49.974222 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:49.974190    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c06fc31_684e_44e6_ab64_0f684831dc54.slice/crio-734fe32a5ec07439de4f82e078870a2c2a5c3c79047cadad00205594ed97ca45 WatchSource:0}: Error finding container 734fe32a5ec07439de4f82e078870a2c2a5c3c79047cadad00205594ed97ca45: Status 404 returned error can't find the container with id 734fe32a5ec07439de4f82e078870a2c2a5c3c79047cadad00205594ed97ca45
Oct 31 03:34:49.975032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.975002    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:34:49.977062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.977025    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:49.977148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.977116    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:49.977195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:49.977160    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:50.244118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:50.243880    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7199-8f517a"]
Oct 31 03:34:50.248858 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:34:50.248779    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9291ca0b_465f_4d80_95d1_774ee938deb1.slice/crio-b12a988a387afbc0833e7988247c6029de68ad183af391364ee84631a4f4957a WatchSource:0}: Error finding container b12a988a387afbc0833e7988247c6029de68ad183af391364ee84631a4f4957a: Status 404 returned error can't find the container with id b12a988a387afbc0833e7988247c6029de68ad183af391364ee84631a4f4957a
Oct 31 03:34:50.948784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:50.948163    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7199-8f517a" event={"ID":"9291ca0b-465f-4d80-95d1-774ee938deb1","Type":"ContainerStarted","Data":"b12a988a387afbc0833e7988247c6029de68ad183af391364ee84631a4f4957a"}
Oct 31 03:34:50.952428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:50.952394    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" event={"ID":"8c06fc31-684e-44e6-ab64-0f684831dc54","Type":"ContainerStarted","Data":"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"}
Oct 31 03:34:50.952610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:50.952478    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" event={"ID":"8c06fc31-684e-44e6-ab64-0f684831dc54","Type":"ContainerStarted","Data":"734fe32a5ec07439de4f82e078870a2c2a5c3c79047cadad00205594ed97ca45"}
Oct 31 03:34:51.071806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:51.071753    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podStartSLOduration=3.071738652 podStartE2EDuration="3.071738652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:51.070727347 +0000 UTC m=+1641.085289081" watchObservedRunningTime="2026-10-31 03:34:51.071738652 +0000 UTC m=+1641.086300360"
Oct 31 03:34:51.467669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:51.467642    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:51.469043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:51.469014    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:51.469182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:51.469071    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:52.468444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:52.468405    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:52.468980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:52.468455    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:52.848026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:52.847945    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:34:52.848026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:52.848012    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:34:53.467616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:53.467559    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:53.467616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:53.467608    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:54.468858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:54.468827    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:54.469496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:54.469410    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:55.467447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:55.467410    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:55.467642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:55.467471    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:56.064147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.064113    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7199-8f517a" event={"ID":"9291ca0b-465f-4d80-95d1-774ee938deb1","Type":"ContainerStarted","Data":"b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5"}
Oct 31 03:34:56.170260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.170155    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7199-8f517a" podStartSLOduration=1.580756634 podStartE2EDuration="7.170140936s" totalImagesPullingTime="5.589384302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:56.16831972 +0000 UTC m=+1646.182881429" watchObservedRunningTime="2026-10-31 03:34:56.170140936 +0000 UTC m=+1646.184702644"
Oct 31 03:34:56.467568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.467512    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:34:56.467796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.467603    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:34:56.715745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.715672    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:34:56.715949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:56.715775    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:34:57.472775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:57.472745    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:57.473212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:57.473191    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:57.474108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:57.474088    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:34:57.730800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:57.730713    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7199-8f517a"]
Oct 31 03:34:57.736372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:57.736343    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7199-8f517a"]
Oct 31 03:34:58.082629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:58.081137    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7199-8f517a" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" containerName="test-curl-7199-8f517a" containerID="cri-o://b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5" gracePeriod=2
Oct 31 03:34:58.128199 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:34:58.128133    5831 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-8f517a\" is forbidden: User \"system:node:ip-10-0-140-194.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-194.us-west-2.compute.internal' and this object" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:34:59.093082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.093043    5831 generic.go:356] "Generic (PLEG): container finished" podID="6925218a-23f2-40b0-8762-c6adc4632afa" containerID="74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769" exitCode=0
Oct 31 03:34:59.093636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.093089    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerDied","Data":"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"}
Oct 31 03:34:59.093730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.093637    5831 scope.go:122] "RemoveContainer" containerID="74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"
Oct 31 03:34:59.095559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.095495    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:59.095636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.095604    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:34:59.095674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:34:59.095651    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:00.199588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.197336    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerStarted","Data":"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"}
Oct 31 03:35:00.446103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.446078    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:35:00.497859 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:00.497796    5831 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-8f517a\" is forbidden: User \"system:node:ip-10-0-140-194.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-194.us-west-2.compute.internal' and this object" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:35:00.564051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.564018    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\" (UniqueName: \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\") pod \"9291ca0b-465f-4d80-95d1-774ee938deb1\" (UID: \"9291ca0b-465f-4d80-95d1-774ee938deb1\") "
Oct 31 03:35:00.566355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.566323    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d" pod "9291ca0b-465f-4d80-95d1-774ee938deb1" (UID: "9291ca0b-465f-4d80-95d1-774ee938deb1"). InnerVolumeSpecName "kube-api-access-lg25d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:00.666053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.666016    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg25d\" (UniqueName: \"kubernetes.io/projected/9291ca0b-465f-4d80-95d1-774ee938deb1-kube-api-access-lg25d\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:00.923616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.923571    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:35:00.923988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.923813    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 03:35:00.923988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:00.923876    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 03:35:01.209275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.209241    5831 generic.go:356] "Generic (PLEG): container finished" podID="9291ca0b-465f-4d80-95d1-774ee938deb1" containerID="b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5" exitCode=137
Oct 31 03:35:01.209835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.209308    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:35:01.209835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.209345    5831 scope.go:122] "RemoveContainer" containerID="b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5"
Oct 31 03:35:01.223001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.222979    5831 scope.go:122] "RemoveContainer" containerID="b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5"
Oct 31 03:35:01.223333 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:01.223312    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5\": container with ID starting with b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5 not found: ID does not exist" containerID="b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5"
Oct 31 03:35:01.223432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.223344    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5"} err="failed to get container status \"b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5\": rpc error: code = NotFound desc = could not find container \"b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5\": container with ID starting with b6cc3566321ad45bfc054d6506cf32035402eb5fcc5449d1189578a7084527d5 not found: ID does not exist"
Oct 31 03:35:01.256751 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:01.256714    5831 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-8f517a\" is forbidden: User \"system:node:ip-10-0-140-194.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-194.us-west-2.compute.internal' and this object" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:35:01.303787 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:01.303742    5831 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7199-8f517a\" is forbidden: User \"system:node:ip-10-0-140-194.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-194.us-west-2.compute.internal' and this object" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" pod="opendatahub/test-curl-7199-8f517a"
Oct 31 03:35:01.923705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.923674    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 03:35:01.923906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:01.923736    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 03:35:02.135470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:02.135437    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" path="/var/lib/kubelet/pods/9291ca0b-465f-4d80-95d1-774ee938deb1/volumes"
Oct 31 03:35:02.924517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:02.924480    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 03:35:02.924994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:02.924572    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 03:35:03.924421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:03.924369    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 03:35:03.924643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:03.924441    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 03:35:04.154394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.154361    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:35:04.155019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.155006    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" containerName="test-curl-7199-8f517a"
Oct 31 03:35:04.155080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.155020    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" containerName="test-curl-7199-8f517a"
Oct 31 03:35:04.155142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.155134    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9291ca0b-465f-4d80-95d1-774ee938deb1" containerName="test-curl-7199-8f517a"
Oct 31 03:35:04.162955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.162935    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.333965    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334023    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334162    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334225    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szr2p\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334356    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334395    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334418    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.334484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.334461    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.378555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.369229    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:35:04.436209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436166    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436230    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436254    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436276    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436317    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436345    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436386    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436447    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.436652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.436486    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szr2p\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.437871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.437811    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.437990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.437916    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.437990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.437948    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.438236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.438203    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.438747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.438723    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.439711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.439694    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.440090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.440073    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.507402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.507367    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.507402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.507388    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szr2p\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.783174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.782810    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:04.930432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.929964    5831 patch_prober.go:29] interesting pod/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 03:35:04.930432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:04.930029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 03:35:05.434840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.428168    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:35:05.518557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.511747    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:05.518557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.511849    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:05.518557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.511900    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:05.929558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.929514    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:35:05.929981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.929962    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:35:05.930787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:05.930696    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:35:06.263410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.263215    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" event={"ID":"b3aaa60b-f502-452f-b96c-354bec49fab7","Type":"ContainerStarted","Data":"22a858bde9571f0694530643bce3b8e2538870594c5b89f295cfce675d1f1253"}
Oct 31 03:35:06.269140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.266145    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" event={"ID":"b3aaa60b-f502-452f-b96c-354bec49fab7","Type":"ContainerStarted","Data":"0d689447573c9cf9b4600058160deb95ed60f611bdb4b39767e2f5426323ab86"}
Oct 31 03:35:06.377313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.377171    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podStartSLOduration=2.377157625 podStartE2EDuration="2.377157625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:06.375910517 +0000 UTC m=+1656.390472229" watchObservedRunningTime="2026-10-31 03:35:06.377157625 +0000 UTC m=+1656.391719335"
Oct 31 03:35:06.787593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.786464    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:06.787821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.787680    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:06.787821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:06.787734    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:07.790556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:07.787984    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:07.790556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:07.788050    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:08.788559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:08.784986    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:08.788559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:08.785087    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:09.804592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:09.804273    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:09.804592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:09.804391    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:10.168207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:10.168121    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:35:10.168379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:10.168313    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy" containerID="cri-o://037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375" gracePeriod=30
Oct 31 03:35:10.791552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:10.783948    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:10.791552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:10.784006    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:11.784428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:11.784393    5831 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 31 03:35:11.784831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:11.784463    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 31 03:35:11.951500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:11.951460    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:35:11.956833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:11.956806    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.007226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.007190    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8b4030-openshift-default-dockercfg-ck5hf\""
Oct 31 03:35:12.012702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.012667    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.013615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.013049    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.013826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.013803    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014025    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjkx\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014250    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014370    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014517    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.014738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.014636    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.059226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.059130    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:35:12.115599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115505    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115616    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115634    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115668    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115691    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjkx\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115713    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.115806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115737    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115808    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.115900    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.116025    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.116144    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.116322    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.116559    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.116754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.116731    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.119360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.119331    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.120124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.120102    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.170312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.170276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.170891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.170867    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjkx\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\") pod \"e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") " pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.282813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.282778    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:12.788998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.788970    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:12.789507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.789486    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:12.790393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.790368    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:35:12.895884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.895814    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:35:12.897809 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:35:12.897777    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5467d59_dc1c_4d02_b4c7_436622d4f971.slice/crio-9592dca161dc4d3997bea6796671a250809c307c645f20f4bff59bfcecbf5f2d WatchSource:0}: Error finding container 9592dca161dc4d3997bea6796671a250809c307c645f20f4bff59bfcecbf5f2d: Status 404 returned error can't find the container with id 9592dca161dc4d3997bea6796671a250809c307c645f20f4bff59bfcecbf5f2d
Oct 31 03:35:12.900939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.900902    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:12.901045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.900986    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:12.901045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:12.901030    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:35:12.992880 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:12.992820    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:35:13.503333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:13.503298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerStarted","Data":"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"}
Oct 31 03:35:13.503569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:13.503349    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerStarted","Data":"9592dca161dc4d3997bea6796671a250809c307c645f20f4bff59bfcecbf5f2d"}
Oct 31 03:35:13.617752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:13.617703    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podStartSLOduration=2.61768523 podStartE2EDuration="2.61768523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:13.613697351 +0000 UTC m=+1663.628259059" watchObservedRunningTime="2026-10-31 03:35:13.61768523 +0000 UTC m=+1663.632246929"
Oct 31 03:35:14.283697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:14.283663    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:14.290322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:14.290298    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:14.510740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:14.510713    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:14.511659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:14.511638    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:35:15.465380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.465355    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:35:15.519760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.519723    5831 generic.go:356] "Generic (PLEG): container finished" podID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerID="037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375" exitCode=0
Oct 31 03:35:15.519954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.519788    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"
Oct 31 03:35:15.519954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.519804    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerDied","Data":"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"}
Oct 31 03:35:15.519954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.519855    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z" event={"ID":"7ae2bd00-09a0-4542-84a7-980bb916168e","Type":"ContainerDied","Data":"bb39164dec7373f05bcac805577bd11b6ee7c3bee6fdbed0588477c3b66375ee"}
Oct 31 03:35:15.519954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.519882    5831 scope.go:122] "RemoveContainer" containerID="037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"
Oct 31 03:35:15.531448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.531430    5831 scope.go:122] "RemoveContainer" containerID="e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"
Oct 31 03:35:15.542159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.542144    5831 scope.go:122] "RemoveContainer" containerID="037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"
Oct 31 03:35:15.542434 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:15.542417    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375\": container with ID starting with 037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375 not found: ID does not exist" containerID="037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"
Oct 31 03:35:15.542513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.542443    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375"} err="failed to get container status \"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375\": rpc error: code = NotFound desc = could not find container \"037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375\": container with ID starting with 037e9f9e194fbd007b95cd044c9751d87dfc83a1c51474182b02d9e1135b6375 not found: ID does not exist"
Oct 31 03:35:15.542513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.542463    5831 scope.go:122] "RemoveContainer" containerID="e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"
Oct 31 03:35:15.542729 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:15.542712    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a\": container with ID starting with e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a not found: ID does not exist" containerID="e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"
Oct 31 03:35:15.542772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.542733    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a"} err="failed to get container status \"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a\": rpc error: code = NotFound desc = could not find container \"e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a\": container with ID starting with e318045b709b476ac9a1e119b6963dad6a7e590b484863e5e19bcda06442ca5a not found: ID does not exist"
Oct 31 03:35:15.579852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.579818    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.579922    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.579977    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580004    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580085    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580141    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580252    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:15.580313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580259    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580359    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580360    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:15.580422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580406    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:15.580602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580487    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\") pod \"7ae2bd00-09a0-4542-84a7-980bb916168e\" (UID: \"7ae2bd00-09a0-4542-84a7-980bb916168e\") "
Oct 31 03:35:15.580790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580750    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:15.580790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.580744    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:15.581223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.581131    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.581223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.581155    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.581223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.581167    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae2bd00-09a0-4542-84a7-980bb916168e-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.581223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.581179    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.581223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.581191    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.582861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.582835    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:15.582967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.582870    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:35:15.583094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.583071    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "kube-api-access-pkkpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:15.583382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.583357    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy" pod "7ae2bd00-09a0-4542-84a7-980bb916168e" (UID: "7ae2bd00-09a0-4542-84a7-980bb916168e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:35:15.683318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.683284    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.683318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.683319    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.683600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.683336    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkpl\" (UniqueName: \"kubernetes.io/projected/7ae2bd00-09a0-4542-84a7-980bb916168e-kube-api-access-pkkpl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.683600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.683352    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae2bd00-09a0-4542-84a7-980bb916168e-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:15.934740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.934706    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:35:15.939508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:15.939480    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2260d8-openshift-default-6cbd66bf99-22j9z"]
Oct 31 03:35:16.137061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:16.136980    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" path="/var/lib/kubelet/pods/7ae2bd00-09a0-4542-84a7-980bb916168e/volumes"
Oct 31 03:35:20.145110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.145075    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:35:20.145965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.145943    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.145965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.145966    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.146125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.145981    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.146125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.145988    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.146234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.146181    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.146234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.146198    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae2bd00-09a0-4542-84a7-980bb916168e" containerName="istio-proxy"
Oct 31 03:35:20.151127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.151107    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.220461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.220426    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:35:20.308111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.297951    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8b4030-dockercfg-zkbm9\""
Oct 31 03:35:20.308111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.298050    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8b4030\""
Oct 31 03:35:20.402868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.402655    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:35:20.447579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.432181    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.462778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.457808    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pn4\" (UniqueName: \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.462778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.457918    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.462778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.457976    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.462778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.458081    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.474978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.474948    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:35:20.559026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.558992    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pn4\" (UniqueName: \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.559194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.559054    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.559194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.559090    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.559194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.559153    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.559870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.559848    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.559950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.559907    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.611820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.611785    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pn4\" (UniqueName: \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.612042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.611863    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\") pod \"payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:20.768074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.768041    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:20.777383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:20.777358    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:21.163126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.163098    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:35:21.163918 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:35:21.163894    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37201e5a_baf9_43e2_97c5_b37bf3c5e9fb.slice/crio-417cf5d59778c30ce283f2e8eb75be3512a3aea817b2876b6224a47d8ca97e94 WatchSource:0}: Error finding container 417cf5d59778c30ce283f2e8eb75be3512a3aea817b2876b6224a47d8ca97e94: Status 404 returned error can't find the container with id 417cf5d59778c30ce283f2e8eb75be3512a3aea817b2876b6224a47d8ca97e94
Oct 31 03:35:21.209907 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:35:21.209878    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d1b43c_813b_4dc6_a892_acb0930e8ff0.slice/crio-fe3fc545410a49602ffce3aa2d8b2702ca0f030e50c7b72424f672eac0cef645 WatchSource:0}: Error finding container fe3fc545410a49602ffce3aa2d8b2702ca0f030e50c7b72424f672eac0cef645: Status 404 returned error can't find the container with id fe3fc545410a49602ffce3aa2d8b2702ca0f030e50c7b72424f672eac0cef645
Oct 31 03:35:21.212091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.212062    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:35:21.325240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.325196    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:35:21.594153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.594114    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" event={"ID":"13d1b43c-813b-4dc6-a892-acb0930e8ff0","Type":"ContainerStarted","Data":"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"}
Oct 31 03:35:21.594368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.594186    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:21.594368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.594214    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" event={"ID":"13d1b43c-813b-4dc6-a892-acb0930e8ff0","Type":"ContainerStarted","Data":"fe3fc545410a49602ffce3aa2d8b2702ca0f030e50c7b72424f672eac0cef645"}
Oct 31 03:35:21.596090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.596060    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" event={"ID":"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb","Type":"ContainerStarted","Data":"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"}
Oct 31 03:35:21.596222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.596100    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" event={"ID":"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb","Type":"ContainerStarted","Data":"417cf5d59778c30ce283f2e8eb75be3512a3aea817b2876b6224a47d8ca97e94"}
Oct 31 03:35:21.596222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.596132    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:21.699900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.699845    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" podStartSLOduration=1.699831189 podStartE2EDuration="1.699831189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:21.699609465 +0000 UTC m=+1671.714171173" watchObservedRunningTime="2026-10-31 03:35:21.699831189 +0000 UTC m=+1671.714392897"
Oct 31 03:35:21.813780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:21.813707    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" podStartSLOduration=1.813687309 podStartE2EDuration="1.813687309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:21.80837119 +0000 UTC m=+1671.822932899" watchObservedRunningTime="2026-10-31 03:35:21.813687309 +0000 UTC m=+1671.828249017"
Oct 31 03:35:22.691766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.691732    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:35:22.696331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.696308    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.791335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.791303    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5d3f8a-serving-cert\""
Oct 31 03:35:22.795570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.795520    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:35:22.798060 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.798038    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.798143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.798095    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.798143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.798116    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.798254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.798237    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msl5\" (UniqueName: \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.899346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.899313    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.899581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.899374    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.899581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.899404    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.899581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.899431    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msl5\" (UniqueName: \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.899581 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:22.899565    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-serving-cert: secret "maas-api-e2e-ipp-a-5d3f8a-serving-cert" not found
Oct 31 03:35:22.899831 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:22.899634    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls podName:7746c1ee-b1e3-4bd3-9882-ab0f9168ae16 nodeName:}" failed. No retries permitted until 2026-10-31 03:35:23.399614191 +0000 UTC m=+1673.414175900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls") pod "maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" (UID: "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16") : secret "maas-api-e2e-ipp-a-5d3f8a-serving-cert" not found
Oct 31 03:35:22.900151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.900126    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.901966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.901941    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:22.982612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:22.982421    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msl5\" (UniqueName: \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:23.420855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:23.420750    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:23.423546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:23.423502    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:23.611289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:23.611247    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:23.993670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:23.993638    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:35:23.995233 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:35:23.995208    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7746c1ee_b1e3_4bd3_9882_ab0f9168ae16.slice/crio-39ad1d2efe1bc729391514baa25aa6e99a40adf7012c4345adda0fd4ef7ef99b WatchSource:0}: Error finding container 39ad1d2efe1bc729391514baa25aa6e99a40adf7012c4345adda0fd4ef7ef99b: Status 404 returned error can't find the container with id 39ad1d2efe1bc729391514baa25aa6e99a40adf7012c4345adda0fd4ef7ef99b
Oct 31 03:35:24.620524 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:24.620491    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" event={"ID":"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16","Type":"ContainerStarted","Data":"39ad1d2efe1bc729391514baa25aa6e99a40adf7012c4345adda0fd4ef7ef99b"}
Oct 31 03:35:25.637962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:25.637915    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" event={"ID":"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16","Type":"ContainerStarted","Data":"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"}
Oct 31 03:35:25.638373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:25.638036    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:25.746706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:25.746654    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" podStartSLOduration=2.793173337 podStartE2EDuration="3.74663672s" totalImagesPullingTime="953.463383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:25.744485102 +0000 UTC m=+1675.759046809" watchObservedRunningTime="2026-10-31 03:35:25.74663672 +0000 UTC m=+1675.761198427"
Oct 31 03:35:28.126083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.126051    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:35:28.130890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.130863    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.184324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.184288    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:35:28.194611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.194578    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.194772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.194665    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnl7n\" (UniqueName: \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.194772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.194707    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.194898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.194841    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.224766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.224734    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e1aad0-serving-cert\""
Oct 31 03:35:28.296087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.296046    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.296260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.296192    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.296260 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:28.296216    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-serving-cert: secret "maas-api-e2e-ipp-a-e1aad0-serving-cert" not found
Oct 31 03:35:28.296260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.296234    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl7n\" (UniqueName: \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.296445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.296264    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.296445 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:28.296283    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls podName:2974f595-4c3d-434a-95e9-95ccf152f9b4 nodeName:}" failed. No retries permitted until 2026-10-31 03:35:28.796263711 +0000 UTC m=+1678.810825401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls") pod "maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" (UID: "2974f595-4c3d-434a-95e9-95ccf152f9b4") : secret "maas-api-e2e-ipp-a-e1aad0-serving-cert" not found
Oct 31 03:35:28.296955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.296928    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.299242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.299212    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.349513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.349480    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl7n\" (UniqueName: \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.808313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.808281    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:28.811165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:28.811136    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:29.047342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:29.047310    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:29.428270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:29.428234    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:35:29.429259 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:35:29.429234    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2974f595_4c3d_434a_95e9_95ccf152f9b4.slice/crio-43bc6f777f663b04ac0903ccaf0136caa0a6a6cf9248a1c6d4fc658329d9f379 WatchSource:0}: Error finding container 43bc6f777f663b04ac0903ccaf0136caa0a6a6cf9248a1c6d4fc658329d9f379: Status 404 returned error can't find the container with id 43bc6f777f663b04ac0903ccaf0136caa0a6a6cf9248a1c6d4fc658329d9f379
Oct 31 03:35:29.672387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:29.672349    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" event={"ID":"2974f595-4c3d-434a-95e9-95ccf152f9b4","Type":"ContainerStarted","Data":"43bc6f777f663b04ac0903ccaf0136caa0a6a6cf9248a1c6d4fc658329d9f379"}
Oct 31 03:35:30.686175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:30.686137    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" event={"ID":"2974f595-4c3d-434a-95e9-95ccf152f9b4","Type":"ContainerStarted","Data":"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"}
Oct 31 03:35:30.686620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:30.686226    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:30.797079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:30.797015    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" podStartSLOduration=1.825188257 podStartE2EDuration="2.796999116s" totalImagesPullingTime="971.810859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:30.79284687 +0000 UTC m=+1680.807408578" watchObservedRunningTime="2026-10-31 03:35:30.796999116 +0000 UTC m=+1680.811560824"
Oct 31 03:35:31.652899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:31.652872    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:35:32.607698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:32.607669    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:35:32.608139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:32.607747    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:35:33.953718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:33.953686    5831 scope.go:122] "RemoveContainer" containerID="43a3609c8688a0d16dc974c782c7ce5ff677983024254a8e8f9b14c2b152b40c"
Oct 31 03:35:33.964699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:33.964676    5831 scope.go:122] "RemoveContainer" containerID="694105843a756a02f2ae1ec178b752708fd59a651c6c06e90e797492c1f46372"
Oct 31 03:35:33.974915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:33.974892    5831 scope.go:122] "RemoveContainer" containerID="212fb4311441a00a900262eb0c403b54f477bf3d9d5ce0b764aa18d008bacecf"
Oct 31 03:35:36.700431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:36.700402    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:35:42.982796 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:42.982740    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:35:43.837071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:43.837035    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:35:43.947953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:43.947909    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:35:48.468732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:48.468693    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:35:48.903617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:48.881946    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" podUID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" containerName="payload-processing" containerID="cri-o://4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3" gracePeriod=25
Oct 31 03:35:48.974551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:48.953621    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" podUID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" containerName="payload-pre-processing" containerID="cri-o://2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147" gracePeriod=25
Oct 31 03:35:49.572931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:49.572901    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:35:49.675980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:49.665546    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\") pod \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") "
Oct 31 03:35:49.675980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:49.665604    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\" (UniqueName: \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\") pod \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\" (UID: \"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b\") "
Oct 31 03:35:49.675980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:49.672003    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume" pod "4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" (UID: "4089ad1f-4e4e-43a1-a4b1-a4bd257b028b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:49.781886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:49.773914    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:50.031575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.017783    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:35:50.031575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.019248    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp" pod "4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" (UID: "4089ad1f-4e4e-43a1-a4b1-a4bd257b028b"). InnerVolumeSpecName "kube-api-access-xfqhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:50.155778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.151214    5831 generic.go:356] "Generic (PLEG): container finished" podID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" containerID="4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3" exitCode=0
Oct 31 03:35:50.158126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.158074    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqhp\" (UniqueName: \"kubernetes.io/projected/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b-kube-api-access-xfqhp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:50.171659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.166258    5831 generic.go:356] "Generic (PLEG): container finished" podID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" containerID="2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147" exitCode=0
Oct 31 03:35:50.274501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.274464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\") pod \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") "
Oct 31 03:35:50.274735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.274516    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\" (UniqueName: \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\") pod \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\" (UID: \"15f4119d-604a-43cf-b9c7-0e9d4c52e086\") "
Oct 31 03:35:50.278851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.278824    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"
Oct 31 03:35:50.295600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.283763    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume" pod "15f4119d-604a-43cf-b9c7-0e9d4c52e086" (UID: "15f4119d-604a-43cf-b9c7-0e9d4c52e086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:50.295600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.283829    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"
Oct 31 03:35:50.342113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.341717    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57" pod "15f4119d-604a-43cf-b9c7-0e9d4c52e086" (UID: "15f4119d-604a-43cf-b9c7-0e9d4c52e086"). InnerVolumeSpecName "kube-api-access-4gv57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:50.377558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.377322    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f4119d-604a-43cf-b9c7-0e9d4c52e086-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:50.377558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.377363    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gv57\" (UniqueName: \"kubernetes.io/projected/15f4119d-604a-43cf-b9c7-0e9d4c52e086-kube-api-access-4gv57\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:50.447007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.441666    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" event={"ID":"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b","Type":"ContainerDied","Data":"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"}
Oct 31 03:35:50.447007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.441792    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk" event={"ID":"4089ad1f-4e4e-43a1-a4b1-a4bd257b028b","Type":"ContainerDied","Data":"937211a927f848d5285fb466ee345d5f0d40cb3fcb103a51c5e94e66c093e762"}
Oct 31 03:35:50.447007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.441810    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" event={"ID":"15f4119d-604a-43cf-b9c7-0e9d4c52e086","Type":"ContainerDied","Data":"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"}
Oct 31 03:35:50.447007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.441831    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb" event={"ID":"15f4119d-604a-43cf-b9c7-0e9d4c52e086","Type":"ContainerDied","Data":"f2514763b4b637a2c1b62a604b91201921f045e8932bc39ff30a0774000da2c7"}
Oct 31 03:35:50.450657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.450624    5831 scope.go:122] "RemoveContainer" containerID="4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"
Oct 31 03:35:50.465517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.465491    5831 scope.go:122] "RemoveContainer" containerID="4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"
Oct 31 03:35:50.465846 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:50.465825    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3\": container with ID starting with 4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3 not found: ID does not exist" containerID="4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"
Oct 31 03:35:50.465919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.465855    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3"} err="failed to get container status \"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3\": rpc error: code = NotFound desc = could not find container \"4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3\": container with ID starting with 4e70a5b654fa0c3fd277c5ab38c7505c208d4378af910ec2548e6bdb4d4971f3 not found: ID does not exist"
Oct 31 03:35:50.465919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.465873    5831 scope.go:122] "RemoveContainer" containerID="2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"
Oct 31 03:35:50.477744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.477694    5831 scope.go:122] "RemoveContainer" containerID="2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"
Oct 31 03:35:50.478208 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:50.478164    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147\": container with ID starting with 2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147 not found: ID does not exist" containerID="2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"
Oct 31 03:35:50.478341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.478200    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147"} err="failed to get container status \"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147\": rpc error: code = NotFound desc = could not find container \"2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147\": container with ID starting with 2bcdfde7330eaebd0f348d1bf92d93218060b067ac1c375d3e52222f1bae2147 not found: ID does not exist"
Oct 31 03:35:50.546788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.546749    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:35:50.554866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.554833    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-5z7bk"]
Oct 31 03:35:50.753623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.753069    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:35:50.758509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:50.758298    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b3a29ec-d9f5dbbc-6xsnb"]
Oct 31 03:35:52.142991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:52.142945    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" path="/var/lib/kubelet/pods/15f4119d-604a-43cf-b9c7-0e9d4c52e086/volumes"
Oct 31 03:35:52.143628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:52.143607    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" path="/var/lib/kubelet/pods/4089ad1f-4e4e-43a1-a4b1-a4bd257b028b/volumes"
Oct 31 03:35:53.470799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.470757    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" podUID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" containerName="payload-pre-processing" containerID="cri-o://7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8" gracePeriod=25
Oct 31 03:35:53.890557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.887747    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:35:53.967820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.967783    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\" (UniqueName: \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\") pod \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") "
Oct 31 03:35:53.968354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.968330    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\") pod \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\" (UID: \"3b56c5a7-b345-46a9-bf9c-27a67c25a64b\") "
Oct 31 03:35:53.968977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.968947    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume" pod "3b56c5a7-b345-46a9-bf9c-27a67c25a64b" (UID: "3b56c5a7-b345-46a9-bf9c-27a67c25a64b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:53.970190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:53.970171    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h" pod "3b56c5a7-b345-46a9-bf9c-27a67c25a64b" (UID: "3b56c5a7-b345-46a9-bf9c-27a67c25a64b"). InnerVolumeSpecName "kube-api-access-kkg4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:54.070368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.070270    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkg4h\" (UniqueName: \"kubernetes.io/projected/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-kube-api-access-kkg4h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:54.070368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.070305    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b56c5a7-b345-46a9-bf9c-27a67c25a64b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:54.259561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.259498    5831 generic.go:356] "Generic (PLEG): container finished" podID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" containerID="7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8" exitCode=0
Oct 31 03:35:54.259714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.259572    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"
Oct 31 03:35:54.259714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.259572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" event={"ID":"3b56c5a7-b345-46a9-bf9c-27a67c25a64b","Type":"ContainerDied","Data":"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"}
Oct 31 03:35:54.259714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.259633    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8" event={"ID":"3b56c5a7-b345-46a9-bf9c-27a67c25a64b","Type":"ContainerDied","Data":"aac93b64eb2f962fd902d891e56b2f5cbe606c8998d86f8eb535fc4bf9ecabf9"}
Oct 31 03:35:54.259714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.259663    5831 scope.go:122] "RemoveContainer" containerID="7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"
Oct 31 03:35:54.272126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.272104    5831 scope.go:122] "RemoveContainer" containerID="7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"
Oct 31 03:35:54.272487 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:35:54.272463    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8\": container with ID starting with 7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8 not found: ID does not exist" containerID="7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"
Oct 31 03:35:54.272583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.272497    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8"} err="failed to get container status \"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8\": rpc error: code = NotFound desc = could not find container \"7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8\": container with ID starting with 7220cc0921374471fa1d2ce965fa8559ef4288fc5ef418d53a88be069b1a55d8 not found: ID does not exist"
Oct 31 03:35:54.415852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.415545    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:35:54.420070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:54.420038    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5effde-578cb47bf5-5njq8"]
Oct 31 03:35:56.138192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:56.138152    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" path="/var/lib/kubelet/pods/3b56c5a7-b345-46a9-bf9c-27a67c25a64b/volumes"
Oct 31 03:35:58.380749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:58.380663    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:35:58.380749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:35:58.380722    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:00.240199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.238586    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240691    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" containerName="payload-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240716    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" containerName="payload-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240757    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" containerName="payload-pre-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240765    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" containerName="payload-pre-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240799    5831 cpu_manager.go:414] "containerMap: removing container" podUID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" containerName="payload-pre-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.240806    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" containerName="payload-pre-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.241242    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f4119d-604a-43cf-b9c7-0e9d4c52e086" containerName="payload-pre-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.241283    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4089ad1f-4e4e-43a1-a4b1-a4bd257b028b" containerName="payload-processing"
Oct 31 03:36:00.242263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.241301    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b56c5a7-b345-46a9-bf9c-27a67c25a64b" containerName="payload-pre-processing"
Oct 31 03:36:00.247123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.247097    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.321224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.321191    5831 generic.go:356] "Generic (PLEG): container finished" podID="d421a5c4-f284-4f4b-8904-41837c250ead" containerID="7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e" exitCode=0
Oct 31 03:36:00.321415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.321267    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerDied","Data":"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"}
Oct 31 03:36:00.321876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.321854    5831 scope.go:122] "RemoveContainer" containerID="7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"
Oct 31 03:36:00.323706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.323664    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.323836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.323737    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.323836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.323774    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.328797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.328774    5831 generic.go:356] "Generic (PLEG): container finished" podID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerID="b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168" exitCode=0
Oct 31 03:36:00.328920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.328869    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerDied","Data":"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"}
Oct 31 03:36:00.329265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.329249    5831 scope.go:122] "RemoveContainer" containerID="b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"
Oct 31 03:36:00.336236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.336201    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.336366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.336276    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.336366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.336336    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:00.362303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362275    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362319    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362487    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362639    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362685    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362789    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59m6\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.362903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.362845    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.363044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.363023    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.363135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.363098    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.438792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.437322    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:36:00.464684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464633    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.464839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464695    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.464839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464749    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59m6\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.464839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464784    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464933    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.464994    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.465076    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.465097    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.465419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.465399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.466091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.466041    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.466091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.466081    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.466291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.466204    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.467495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.467468    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.474861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.474834    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.476729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.476707    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.532390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.532279    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.536680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.536483    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59m6\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:00.602090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:00.601811    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:01.205546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.205491    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:36:01.211124 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:01.211087    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97432e9c_c79c_4bab_8cba_d9aa0eb475d3.slice/crio-5fe8d803b8aba992d78047a5cbefe906b0779ef127d0190d02b8e0468423274f WatchSource:0}: Error finding container 5fe8d803b8aba992d78047a5cbefe906b0779ef127d0190d02b8e0468423274f: Status 404 returned error can't find the container with id 5fe8d803b8aba992d78047a5cbefe906b0779ef127d0190d02b8e0468423274f
Oct 31 03:36:01.214280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.214244    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:01.214391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.214323    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:01.214391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.214369    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:01.350002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.348882    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerStarted","Data":"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"}
Oct 31 03:36:01.356549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.353180    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerStarted","Data":"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"}
Oct 31 03:36:01.356549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.356213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" event={"ID":"97432e9c-c79c-4bab-8cba-d9aa0eb475d3","Type":"ContainerStarted","Data":"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"}
Oct 31 03:36:01.356549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.356289    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" event={"ID":"97432e9c-c79c-4bab-8cba-d9aa0eb475d3","Type":"ContainerStarted","Data":"5fe8d803b8aba992d78047a5cbefe906b0779ef127d0190d02b8e0468423274f"}
Oct 31 03:36:01.604607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.603616    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:01.654677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:01.654624    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podStartSLOduration=1.654610367 podStartE2EDuration="1.654610367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:01.652894132 +0000 UTC m=+1711.667455843" watchObservedRunningTime="2026-10-31 03:36:01.654610367 +0000 UTC m=+1711.669172074"
Oct 31 03:36:02.108554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.104573    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:36:02.108554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.104699    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:02.108554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.104739    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:02.341555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.311959    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:36:02.341555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.312247    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:02.341555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.312292    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:02.400340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.400259    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:36:02.655646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.655523    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:02.656103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:02.656076    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:03.107557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.094897    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:03.107557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.094976    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:03.314176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.314137    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:03.314520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.314491    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:03.609942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.609909    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:03.610870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:03.610816    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:04.094943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.094904    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:04.095939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.095908    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:04.316554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.316026    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:04.316554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.316105    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:04.607554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.603966    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:04.607554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:04.604029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:05.089936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.089899    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:05.090575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.090519    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:05.315957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.315916    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:05.316256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.316217    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:05.695736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.690657    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:05.695736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.690719    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:05.818704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:05.768854    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:36:06.091922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.091836    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:06.092619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.092485    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:06.328725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.328643    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:06.329070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.329038    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:06.604967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.604930    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:06.605267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:06.605235    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:07.104558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.094924    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:36:07.104558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.094986    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:36:07.316931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.316891    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:36:07.317217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.317186    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 03:36:07.580705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.580653    5831 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:07.610000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.609959    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:07.610284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:07.610255    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:08.113557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.112794    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:36:08.116162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.116134    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:36:08.331865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.331835    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:36:08.332832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.332810    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:36:08.333973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.333958    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:36:08.619950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.619909    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:36:08.620164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:08.619991    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:36:09.608360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:09.608328    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:09.608877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:09.608823    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:09.609596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:09.609573    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:36:12.979429 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:12.979388    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:36:13.290794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.290721    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:36:13.295477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.295456    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.343766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.343735    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8b4030-serving-cert\""
Oct 31 03:36:13.383164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.383132    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.383341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.383174    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.383341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.383288    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.383341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.383316    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmbd\" (UniqueName: \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.393298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.393275    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:36:13.484367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.484321    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.484580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.484376    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.484580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.484451    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.484580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.484489    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmbd\" (UniqueName: \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.484988 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:13.484967    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-serving-cert: secret "maas-api-e2e-ipp-a-8b4030-serving-cert" not found
Oct 31 03:36:13.485107 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:13.485035    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls podName:dcfdc610-fc63-4169-9721-9c7557bf0f51 nodeName:}" failed. No retries permitted until 2026-10-31 03:36:13.985020529 +0000 UTC m=+1723.999582216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls") pod "maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" (UID: "dcfdc610-fc63-4169-9721-9c7557bf0f51") : secret "maas-api-e2e-ipp-a-8b4030-serving-cert" not found
Oct 31 03:36:13.485786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.485747    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.488906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.488887    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.539175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.539149    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmbd\" (UniqueName: \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:13.997726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:13.997688    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:14.000315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:14.000287    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:14.211609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:14.211577    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:14.595004 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:14.594951    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:36:14.596865 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:14.596833    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfdc610_fc63_4169_9721_9c7557bf0f51.slice/crio-12dd847ff7029f282844de405bf7da295bb5434543995832ba46a9aa538376c2 WatchSource:0}: Error finding container 12dd847ff7029f282844de405bf7da295bb5434543995832ba46a9aa538376c2: Status 404 returned error can't find the container with id 12dd847ff7029f282844de405bf7da295bb5434543995832ba46a9aa538376c2
Oct 31 03:36:14.911054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:14.910961    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" event={"ID":"dcfdc610-fc63-4169-9721-9c7557bf0f51","Type":"ContainerStarted","Data":"12dd847ff7029f282844de405bf7da295bb5434543995832ba46a9aa538376c2"}
Oct 31 03:36:15.921002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:15.920908    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" event={"ID":"dcfdc610-fc63-4169-9721-9c7557bf0f51","Type":"ContainerStarted","Data":"00468f8327fad82eb8d8e044cea03395c8b41f9487c45f052b912300ef418f12"}
Oct 31 03:36:15.921479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:15.921042    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:16.027456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:16.027403    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" podStartSLOduration=2.196669621 podStartE2EDuration="3.027385317s" totalImagesPullingTime="830.715696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:16.026739132 +0000 UTC m=+1726.041300867" watchObservedRunningTime="2026-10-31 03:36:16.027385317 +0000 UTC m=+1726.041947025"
Oct 31 03:36:16.460734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:16.460694    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:36:16.576105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:16.576071    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:36:21.461555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.461485    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" podUID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" containerName="payload-processing" containerID="cri-o://4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc" gracePeriod=25
Oct 31 03:36:21.577641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.577596    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" podUID="d379b516-a028-49b2-8e4b-e9637de37eea" containerName="payload-pre-processing" containerID="cri-o://77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88" gracePeriod=25
Oct 31 03:36:21.778358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.778331    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:36:21.835032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.835001    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\" (UniqueName: \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\") pod \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") "
Oct 31 03:36:21.835218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.835056    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\") pod \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\" (UID: \"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246\") "
Oct 31 03:36:21.835610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.835571    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume" pod "cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" (UID: "cef5f7f0-03ff-4e6e-b451-39b3b4ebb246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:21.835764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.835708    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:21.838480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.838451    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7" pod "cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" (UID: "cef5f7f0-03ff-4e6e-b451-39b3b4ebb246"). InnerVolumeSpecName "kube-api-access-ldvh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:21.871984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.871961    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:36:21.936418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.936366    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\") pod \"d379b516-a028-49b2-8e4b-e9637de37eea\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") "
Oct 31 03:36:21.936641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.936465    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\" (UniqueName: \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\") pod \"d379b516-a028-49b2-8e4b-e9637de37eea\" (UID: \"d379b516-a028-49b2-8e4b-e9637de37eea\") "
Oct 31 03:36:21.936641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.936494    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:36:21.936902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.936878    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume" pod "d379b516-a028-49b2-8e4b-e9637de37eea" (UID: "d379b516-a028-49b2-8e4b-e9637de37eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:21.937104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.937087    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d379b516-a028-49b2-8e4b-e9637de37eea-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:21.937272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.937252    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvh7\" (UniqueName: \"kubernetes.io/projected/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246-kube-api-access-ldvh7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:21.940119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.940090    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf" pod "d379b516-a028-49b2-8e4b-e9637de37eea" (UID: "d379b516-a028-49b2-8e4b-e9637de37eea"). InnerVolumeSpecName "kube-api-access-flgdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:21.974765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.974737    5831 generic.go:356] "Generic (PLEG): container finished" podID="d379b516-a028-49b2-8e4b-e9637de37eea" containerID="77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88" exitCode=0
Oct 31 03:36:21.974911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.974779    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"
Oct 31 03:36:21.974911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.974822    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" event={"ID":"d379b516-a028-49b2-8e4b-e9637de37eea","Type":"ContainerDied","Data":"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"}
Oct 31 03:36:21.974911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.974885    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs" event={"ID":"d379b516-a028-49b2-8e4b-e9637de37eea","Type":"ContainerDied","Data":"fa3d843ade9292528edf8c46abec686380f8a6e0c56b1f0a8cc05870ba20cfbf"}
Oct 31 03:36:21.975082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.974914    5831 scope.go:122] "RemoveContainer" containerID="77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"
Oct 31 03:36:21.976620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.976595    5831 generic.go:356] "Generic (PLEG): container finished" podID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" containerID="4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc" exitCode=0
Oct 31 03:36:21.976740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.976633    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"
Oct 31 03:36:21.976740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.976636    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" event={"ID":"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246","Type":"ContainerDied","Data":"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"}
Oct 31 03:36:21.976823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:21.976740    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s" event={"ID":"cef5f7f0-03ff-4e6e-b451-39b3b4ebb246","Type":"ContainerDied","Data":"01e61ba21b9946f321b26bbd03f2d4d0efa3b980f71b51b38b564297add3f198"}
Oct 31 03:36:22.006908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.006877    5831 scope.go:122] "RemoveContainer" containerID="77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"
Oct 31 03:36:22.007273 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:22.007247    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88\": container with ID starting with 77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88 not found: ID does not exist" containerID="77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"
Oct 31 03:36:22.007380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.007291    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88"} err="failed to get container status \"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88\": rpc error: code = NotFound desc = could not find container \"77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88\": container with ID starting with 77429c8b8902145d990ab38b72a4aa77696604789abd8aefcc8cfff3294cde88 not found: ID does not exist"
Oct 31 03:36:22.007380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.007320    5831 scope.go:122] "RemoveContainer" containerID="4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"
Oct 31 03:36:22.022719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.022696    5831 scope.go:122] "RemoveContainer" containerID="4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"
Oct 31 03:36:22.023091 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:22.023061    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc\": container with ID starting with 4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc not found: ID does not exist" containerID="4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"
Oct 31 03:36:22.023170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.023103    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc"} err="failed to get container status \"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc\": rpc error: code = NotFound desc = could not find container \"4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc\": container with ID starting with 4645fbbed3cbec3bce0cc9c8fa94d809a271b46b9bf2a391c341610f20831afc not found: ID does not exist"
Oct 31 03:36:22.038997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.038967    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgdf\" (UniqueName: \"kubernetes.io/projected/d379b516-a028-49b2-8e4b-e9637de37eea-kube-api-access-flgdf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:22.170274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.170220    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:36:22.173957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.173932    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87af9c-789b5d6fb9-snw9s"]
Oct 31 03:36:22.275543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.275443    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:36:22.280840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:22.280136    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87af9c-7bc97665f9-pghvs"]
Oct 31 03:36:24.044866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:24.044826    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"]
Oct 31 03:36:24.098578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:24.096340    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b7446d-52mtr"]
Oct 31 03:36:24.137412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:24.137373    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccabe6f4-34b4-4861-9972-31270f6777da" path="/var/lib/kubelet/pods/ccabe6f4-34b4-4861-9972-31270f6777da/volumes"
Oct 31 03:36:24.137895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:24.137878    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" path="/var/lib/kubelet/pods/cef5f7f0-03ff-4e6e-b451-39b3b4ebb246/volumes"
Oct 31 03:36:24.138280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:24.138265    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d379b516-a028-49b2-8e4b-e9637de37eea" path="/var/lib/kubelet/pods/d379b516-a028-49b2-8e4b-e9637de37eea/volumes"
Oct 31 03:36:33.609597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.609560    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:36:33.610333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610315    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d379b516-a028-49b2-8e4b-e9637de37eea" containerName="payload-pre-processing"
Oct 31 03:36:33.610396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610334    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d379b516-a028-49b2-8e4b-e9637de37eea" containerName="payload-pre-processing"
Oct 31 03:36:33.610396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610351    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" containerName="payload-processing"
Oct 31 03:36:33.610396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610356    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" containerName="payload-processing"
Oct 31 03:36:33.610575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610472    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d379b516-a028-49b2-8e4b-e9637de37eea" containerName="payload-pre-processing"
Oct 31 03:36:33.610575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.610488    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef5f7f0-03ff-4e6e-b451-39b3b4ebb246" containerName="payload-processing"
Oct 31 03:36:33.615914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.615894    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.673560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.671154    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-18d5f6-openshift-default-dockercfg-hj9qn\""
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759312    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759415    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759445    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759478    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759615    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759691    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759727    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759755    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.759783    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89n6\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.761381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.760064    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887368    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887426    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887463    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887672    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887709    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887738    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887771    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89n6\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.887885    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.888404    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.888684    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.889263    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.914968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.914949    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.917171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.917139    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:33.956907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:33.952865    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:34.003469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.003388    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:34.013346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.013307    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89n6\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:34.016055 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.016023    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\") pod \"e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") " pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:34.055555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.054955    5831 scope.go:122] "RemoveContainer" containerID="24b32962732c73b8131f32cf79b2d084f3fe6a3ddcf379bad31c905c5e4bb13a"
Oct 31 03:36:34.143663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.143631    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:36:34.201451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.201420    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:36:34.241129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.241097    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:34.757035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.756999    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:36:34.795958 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:34.795913    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96154984_ffc5_48e8_a744_6032cddc2a00.slice/crio-a0c5f786372743d4ba1fe43a266c97fc978d7768f08bcb21405b8a0f2406ed57 WatchSource:0}: Error finding container a0c5f786372743d4ba1fe43a266c97fc978d7768f08bcb21405b8a0f2406ed57: Status 404 returned error can't find the container with id a0c5f786372743d4ba1fe43a266c97fc978d7768f08bcb21405b8a0f2406ed57
Oct 31 03:36:34.798944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.798908    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:34.799032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.799019    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:34.799076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:34.799053    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:35.003389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.003327    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:36:35.003916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.003882    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" podUID="cf88afca-7746-4233-8c4e-98a3b2d0131d" containerName="maas-api" containerID="cri-o://a07e3c2df9001e1db34f603992983f56daa011c8a5339e22deebb3bd1546cded" gracePeriod=30
Oct 31 03:36:35.197708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.191378    5831 generic.go:356] "Generic (PLEG): container finished" podID="cf88afca-7746-4233-8c4e-98a3b2d0131d" containerID="a07e3c2df9001e1db34f603992983f56daa011c8a5339e22deebb3bd1546cded" exitCode=0
Oct 31 03:36:35.197708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.191547    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" event={"ID":"cf88afca-7746-4233-8c4e-98a3b2d0131d","Type":"ContainerDied","Data":"a07e3c2df9001e1db34f603992983f56daa011c8a5339e22deebb3bd1546cded"}
Oct 31 03:36:35.203132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.201965    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerStarted","Data":"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"}
Oct 31 03:36:35.203132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.202030    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerStarted","Data":"a0c5f786372743d4ba1fe43a266c97fc978d7768f08bcb21405b8a0f2406ed57"}
Oct 31 03:36:35.346820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.333363    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podStartSLOduration=2.33333798 podStartE2EDuration="2.33333798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:35.332030724 +0000 UTC m=+1745.346592449" watchObservedRunningTime="2026-10-31 03:36:35.33333798 +0000 UTC m=+1745.347899712"
Oct 31 03:36:35.537554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.521959    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:35.676914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.616677    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:36:35.676914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:35.617276    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:36:36.104757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.104724    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:36:36.227821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.224889    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") pod \"cf88afca-7746-4233-8c4e-98a3b2d0131d\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") "
Oct 31 03:36:36.227821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.225005    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\" (UniqueName: \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\") pod \"cf88afca-7746-4233-8c4e-98a3b2d0131d\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") "
Oct 31 03:36:36.227821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.225054    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\") pod \"cf88afca-7746-4233-8c4e-98a3b2d0131d\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") "
Oct 31 03:36:36.227821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.225103    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\") pod \"cf88afca-7746-4233-8c4e-98a3b2d0131d\" (UID: \"cf88afca-7746-4233-8c4e-98a3b2d0131d\") "
Oct 31 03:36:36.227821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.226345    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca" pod "cf88afca-7746-4233-8c4e-98a3b2d0131d" (UID: "cf88afca-7746-4233-8c4e-98a3b2d0131d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:36.284020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.268918    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs" pod "cf88afca-7746-4233-8c4e-98a3b2d0131d" (UID: "cf88afca-7746-4233-8c4e-98a3b2d0131d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:36.300586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.292063    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:36.345187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.335460    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:36.345187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.335497    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf88afca-7746-4233-8c4e-98a3b2d0131d-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:36.457554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.445513    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"
Oct 31 03:36:36.457554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.445788    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr" event={"ID":"cf88afca-7746-4233-8c4e-98a3b2d0131d","Type":"ContainerDied","Data":"b8b834952406db23cb9e023b8812262a452618bad5f70787fe3aceaa6b6a5bd9"}
Oct 31 03:36:36.457554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.445849    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:36.457554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.445911    5831 scope.go:122] "RemoveContainer" containerID="a07e3c2df9001e1db34f603992983f56daa011c8a5339e22deebb3bd1546cded"
Oct 31 03:36:36.464402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.463988    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr" pod "cf88afca-7746-4233-8c4e-98a3b2d0131d" (UID: "cf88afca-7746-4233-8c4e-98a3b2d0131d"). InnerVolumeSpecName "kube-api-access-5bhlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:36.464734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.464688    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls" pod "cf88afca-7746-4233-8c4e-98a3b2d0131d" (UID: "cf88afca-7746-4233-8c4e-98a3b2d0131d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:36.487639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.487015    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:36:36.546751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.546608    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf88afca-7746-4233-8c4e-98a3b2d0131d-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:36.546751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.546643    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhlr\" (UniqueName: \"kubernetes.io/projected/cf88afca-7746-4233-8c4e-98a3b2d0131d-kube-api-access-5bhlr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:36.862778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.862737    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:36:36.866503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:36.866476    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87af9c-565d99cbc8-vdzdr"]
Oct 31 03:36:38.137676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:38.137637    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf88afca-7746-4233-8c4e-98a3b2d0131d" path="/var/lib/kubelet/pods/cf88afca-7746-4233-8c4e-98a3b2d0131d/volumes"
Oct 31 03:36:39.150619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.148602    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" podUID="54b024c5-a49b-4e08-a533-44f0656e8aef" containerName="payload-processing" containerID="cri-o://9e0c59ce2ecc2051625166b6e3a78a6cccbb05b2f7f527084fed1941c49f3534" gracePeriod=25
Oct 31 03:36:39.211623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.210636    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" podUID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" containerName="payload-pre-processing" containerID="cri-o://fa00d3b342dde22ac36dbe66282c0e892028b6babc3ba3ba0abafad4c49c6979" gracePeriod=25
Oct 31 03:36:39.371575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.371507    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:36:39.372501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.372473    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cf88afca-7746-4233-8c4e-98a3b2d0131d" containerName="maas-api"
Oct 31 03:36:39.372501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.372502    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf88afca-7746-4233-8c4e-98a3b2d0131d" containerName="maas-api"
Oct 31 03:36:39.372740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.372673    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf88afca-7746-4233-8c4e-98a3b2d0131d" containerName="maas-api"
Oct 31 03:36:39.377093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.377072    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.431432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.431227    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5d3964-openshift-default-dockercfg-ppwjx\""
Oct 31 03:36:39.492327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.492278    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:36:39.493594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.493565    5831 generic.go:356] "Generic (PLEG): container finished" podID="54b024c5-a49b-4e08-a533-44f0656e8aef" containerID="9e0c59ce2ecc2051625166b6e3a78a6cccbb05b2f7f527084fed1941c49f3534" exitCode=0
Oct 31 03:36:39.493858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.493838    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" event={"ID":"54b024c5-a49b-4e08-a533-44f0656e8aef","Type":"ContainerDied","Data":"9e0c59ce2ecc2051625166b6e3a78a6cccbb05b2f7f527084fed1941c49f3534"}
Oct 31 03:36:39.497304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.497278    5831 generic.go:356] "Generic (PLEG): container finished" podID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerID="aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72" exitCode=0
Oct 31 03:36:39.497556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.497519    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerDied","Data":"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"}
Oct 31 03:36:39.499363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.499340    5831 scope.go:122] "RemoveContainer" containerID="aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"
Oct 31 03:36:39.502288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502085    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502127    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502163    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502248    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502286    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502317    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502352    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502413    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.502644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.502466    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lgm\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.504139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.504046    5831 generic.go:356] "Generic (PLEG): container finished" podID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" containerID="fa00d3b342dde22ac36dbe66282c0e892028b6babc3ba3ba0abafad4c49c6979" exitCode=0
Oct 31 03:36:39.504139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.504080    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" event={"ID":"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b","Type":"ContainerDied","Data":"fa00d3b342dde22ac36dbe66282c0e892028b6babc3ba3ba0abafad4c49c6979"}
Oct 31 03:36:39.504392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.504012    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:39.504392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.504363    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:39.504851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.504828    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:39.637466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.629205    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.666548    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lgm\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.666637    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667004    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667057    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667218    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667422    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667495    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667574    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.667624    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.671869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.668048    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.675649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.674373    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.675649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.659981    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.675649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.675372    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.681240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.680851    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.683685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.682220    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.727646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.727615    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lgm\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.727775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.727617    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\") pod \"e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") " pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:39.820333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.820301    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:36:39.827988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.827962    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:36:39.973126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973036    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\" (UniqueName: \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\") pod \"54b024c5-a49b-4e08-a533-44f0656e8aef\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") "
Oct 31 03:36:39.973126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973089    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\") pod \"54b024c5-a49b-4e08-a533-44f0656e8aef\" (UID: \"54b024c5-a49b-4e08-a533-44f0656e8aef\") "
Oct 31 03:36:39.973126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973112    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\") pod \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") "
Oct 31 03:36:39.973443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973233    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\" (UniqueName: \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\") pod \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\" (UID: \"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b\") "
Oct 31 03:36:39.973583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973512    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume" pod "1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" (UID: "1dc7a8b9-92db-447b-b9e5-11beaf7ea97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:39.973654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.973509    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume" pod "54b024c5-a49b-4e08-a533-44f0656e8aef" (UID: "54b024c5-a49b-4e08-a533-44f0656e8aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:39.974265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.974210    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b024c5-a49b-4e08-a533-44f0656e8aef-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:39.974265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.974242    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:39.976269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.976239    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr" pod "54b024c5-a49b-4e08-a533-44f0656e8aef" (UID: "54b024c5-a49b-4e08-a533-44f0656e8aef"). InnerVolumeSpecName "kube-api-access-q5rnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:39.976373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.976348    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5" pod "1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" (UID: "1dc7a8b9-92db-447b-b9e5-11beaf7ea97b"). InnerVolumeSpecName "kube-api-access-tqpp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:39.994470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:39.994444    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:40.079278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.079213    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5rnr\" (UniqueName: \"kubernetes.io/projected/54b024c5-a49b-4e08-a533-44f0656e8aef-kube-api-access-q5rnr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:40.079278 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.079253    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqpp5\" (UniqueName: \"kubernetes.io/projected/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b-kube-api-access-tqpp5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:40.427197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.427169    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:36:40.428722 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:40.428697    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea37d7c_17bd_4875_b050_25f93945eed1.slice/crio-345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e WatchSource:0}: Error finding container 345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e: Status 404 returned error can't find the container with id 345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e
Oct 31 03:36:40.431426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.431396    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:40.431522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.431462    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:40.431522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.431495    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:40.521116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.521087    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"
Oct 31 03:36:40.521312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.521086    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj" event={"ID":"54b024c5-a49b-4e08-a533-44f0656e8aef","Type":"ContainerDied","Data":"bbc026101a3c87abde9e916c65f9c474948fd7e20f25893cd19c0e84a4809204"}
Oct 31 03:36:40.521312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.521224    5831 scope.go:122] "RemoveContainer" containerID="9e0c59ce2ecc2051625166b6e3a78a6cccbb05b2f7f527084fed1941c49f3534"
Oct 31 03:36:40.523442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.523421    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerStarted","Data":"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"}
Oct 31 03:36:40.525330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.525304    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"
Oct 31 03:36:40.525441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.525346    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn" event={"ID":"1dc7a8b9-92db-447b-b9e5-11beaf7ea97b","Type":"ContainerDied","Data":"fa2bc48ec4c4670df05a23ba3106f21bfe426c067772ffcb6e2c16d587fa8020"}
Oct 31 03:36:40.526925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.526895    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerStarted","Data":"345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e"}
Oct 31 03:36:40.536343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.536322    5831 scope.go:122] "RemoveContainer" containerID="fa00d3b342dde22ac36dbe66282c0e892028b6babc3ba3ba0abafad4c49c6979"
Oct 31 03:36:40.676967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.676879    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:36:40.705550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.705230    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5767b6cb87-t4htn"]
Oct 31 03:36:40.922001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.921963    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:36:40.928872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:40.928829    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-545bf9dc9-qsdqj"]
Oct 31 03:36:41.135863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.135833    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:36:41.136335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.136301    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy" containerID="cri-o://82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079" gracePeriod=30
Oct 31 03:36:41.305690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.305602    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:36:41.306124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.306100    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:41.306374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.306333    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:41.477588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.460747    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:36:41.560389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.560287    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:36:41.574163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.574130    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerStarted","Data":"0c632bfc639b1a5dec6b455e3f3a1c6209a0bc465f30f10890df23bcfd30e331"}
Oct 31 03:36:41.837833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.837708    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podStartSLOduration=2.8376880030000002 podStartE2EDuration="2.837688003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:41.835042223 +0000 UTC m=+1751.849603950" watchObservedRunningTime="2026-10-31 03:36:41.837688003 +0000 UTC m=+1751.852249712"
Oct 31 03:36:42.002835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:41.999957    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:42.016554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.014922    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:42.201883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.197013    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" path="/var/lib/kubelet/pods/1dc7a8b9-92db-447b-b9e5-11beaf7ea97b/volumes"
Oct 31 03:36:42.201883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.197715    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b024c5-a49b-4e08-a533-44f0656e8aef" path="/var/lib/kubelet/pods/54b024c5-a49b-4e08-a533-44f0656e8aef/volumes"
Oct 31 03:36:42.310374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.310322    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:42.310611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.310384    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:42.613483 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.613386    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:42.614397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.614379    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:36:42.636327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.636295    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:36:42.637497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637470    5831 cpu_manager.go:414] "containerMap: removing container" podUID="54b024c5-a49b-4e08-a533-44f0656e8aef" containerName="payload-processing"
Oct 31 03:36:42.637497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637496    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b024c5-a49b-4e08-a533-44f0656e8aef" containerName="payload-processing"
Oct 31 03:36:42.637682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637548    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" containerName="payload-pre-processing"
Oct 31 03:36:42.637682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637556    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" containerName="payload-pre-processing"
Oct 31 03:36:42.637802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637731    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc7a8b9-92db-447b-b9e5-11beaf7ea97b" containerName="payload-pre-processing"
Oct 31 03:36:42.637802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.637769    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b024c5-a49b-4e08-a533-44f0656e8aef" containerName="payload-processing"
Oct 31 03:36:42.643111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.643094    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.768560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.751840    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a32cdb-openshift-default-dockercfg-b5nrg\""
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823013    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823125    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823184    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823210    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823337    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlzm\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823364    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823440    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823523    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.823866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.823583    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlzm\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945434    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945502    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945643    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945706    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945795    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945843    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.945871    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.946404    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.969131    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.969990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.970244    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.972558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.970484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:42.993129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:42.973652    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:36:43.273633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.234715    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlzm\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:43.306517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.305181    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:43.329468 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:43.328216    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:36:43.348568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.337490    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:43.367691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.367655    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\") pod \"e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") " pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:43.402581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.398998    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:43.402581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.399131    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:43.571311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:43.571222    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:44.276946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.276913    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:36:44.277817 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:44.277784    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dddabc_e18b_4d28_8dcc_ff1a774979bd.slice/crio-8bde45454dc75715ee12569da09b86855c88d3250feb23d6b0887cd75bbfcef0 WatchSource:0}: Error finding container 8bde45454dc75715ee12569da09b86855c88d3250feb23d6b0887cd75bbfcef0: Status 404 returned error can't find the container with id 8bde45454dc75715ee12569da09b86855c88d3250feb23d6b0887cd75bbfcef0
Oct 31 03:36:44.295738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.293007    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:44.295738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.293053    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:44.295738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.293358    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:44.295738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.293444    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:44.295738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.293487    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:36:44.516612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.512620    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:36:44.711901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.711866    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerStarted","Data":"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"}
Oct 31 03:36:44.712117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.711926    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerStarted","Data":"8bde45454dc75715ee12569da09b86855c88d3250feb23d6b0887cd75bbfcef0"}
Oct 31 03:36:44.821140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:44.821050    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podStartSLOduration=2.821036786 podStartE2EDuration="2.821036786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:44.819871063 +0000 UTC m=+1754.834432752" watchObservedRunningTime="2026-10-31 03:36:44.821036786 +0000 UTC m=+1754.835598495"
Oct 31 03:36:45.299555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.283990    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:45.299555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.284069    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:45.594689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.594585    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:45.610556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.609807    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:45.724277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.724135    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:45.725416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:45.725390    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:36:46.284662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.284625    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 31 03:36:46.284866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.284678    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 31 03:36:46.444447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.444418    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:36:46.479372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.479303    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" podUID="4210d2d9-981a-4e07-a275-4b5e887d70b4" containerName="payload-processing" containerID="cri-o://0b938a2ae2fe88a5a7da72c9e3766dd2b9b631c209b6373fb647fed037cef269" gracePeriod=25
Oct 31 03:36:46.561423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.561387    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" podUID="d38261dc-5682-45d8-89a9-2320e5196831" containerName="payload-pre-processing" containerID="cri-o://c440897344225ebb874eff1dbb2a410591f924022f5f7ce3fc7f39349f3cf88e" gracePeriod=25
Oct 31 03:36:46.561781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.561758    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.561892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.561847    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.561892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.561887    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.561962    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562028    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562068    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562110    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:46.562175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562133    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562188    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562303    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\") pod \"6925218a-23f2-40b0-8762-c6adc4632afa\" (UID: \"6925218a-23f2-40b0-8762-c6adc4632afa\") "
Oct 31 03:36:46.562487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562401    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:46.562899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562662    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:46.562899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562681    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:46.562899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.562867    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:46.563265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.563129    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925218a-23f2-40b0-8762-c6adc4632afa-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.563265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.563153    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.563265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.563167    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.563265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.563182    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.564879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.564858    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:46.565504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.565480    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "kube-api-access-pv5rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:46.565653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.565563    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:36:46.566779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.566749    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy" pod "6925218a-23f2-40b0-8762-c6adc4632afa" (UID: "6925218a-23f2-40b0-8762-c6adc4632afa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:46.664607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.664497    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.664607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.664548    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925218a-23f2-40b0-8762-c6adc4632afa-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.664607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.664559    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925218a-23f2-40b0-8762-c6adc4632afa-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.664607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.664568    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.664607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.664578    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5rc\" (UniqueName: \"kubernetes.io/projected/6925218a-23f2-40b0-8762-c6adc4632afa-kube-api-access-pv5rc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.738099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.738061    5831 generic.go:356] "Generic (PLEG): container finished" podID="6925218a-23f2-40b0-8762-c6adc4632afa" containerID="82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079" exitCode=0
Oct 31 03:36:46.738339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.738135    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"
Oct 31 03:36:46.738339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.738152    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerDied","Data":"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"}
Oct 31 03:36:46.738339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.738223    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k" event={"ID":"6925218a-23f2-40b0-8762-c6adc4632afa","Type":"ContainerDied","Data":"f4e241068d3d8dcf3c3d4b67d3a554ed9661a07ff92055ad47236cc0026983ea"}
Oct 31 03:36:46.738339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.738262    5831 scope.go:122] "RemoveContainer" containerID="82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"
Oct 31 03:36:46.744368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.744315    5831 generic.go:356] "Generic (PLEG): container finished" podID="d38261dc-5682-45d8-89a9-2320e5196831" containerID="c440897344225ebb874eff1dbb2a410591f924022f5f7ce3fc7f39349f3cf88e" exitCode=0
Oct 31 03:36:46.744661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.744590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" event={"ID":"d38261dc-5682-45d8-89a9-2320e5196831","Type":"ContainerDied","Data":"c440897344225ebb874eff1dbb2a410591f924022f5f7ce3fc7f39349f3cf88e"}
Oct 31 03:36:46.748823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.747428    5831 generic.go:356] "Generic (PLEG): container finished" podID="4210d2d9-981a-4e07-a275-4b5e887d70b4" containerID="0b938a2ae2fe88a5a7da72c9e3766dd2b9b631c209b6373fb647fed037cef269" exitCode=0
Oct 31 03:36:46.748823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.747593    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" event={"ID":"4210d2d9-981a-4e07-a275-4b5e887d70b4","Type":"ContainerDied","Data":"0b938a2ae2fe88a5a7da72c9e3766dd2b9b631c209b6373fb647fed037cef269"}
Oct 31 03:36:46.766226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.766147    5831 scope.go:122] "RemoveContainer" containerID="74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"
Oct 31 03:36:46.783674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.783648    5831 scope.go:122] "RemoveContainer" containerID="82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"
Oct 31 03:36:46.784282 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:46.784252    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079\": container with ID starting with 82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079 not found: ID does not exist" containerID="82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"
Oct 31 03:36:46.786708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.784298    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079"} err="failed to get container status \"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079\": rpc error: code = NotFound desc = could not find container \"82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079\": container with ID starting with 82d1667cf34acd654e7d71ad742e9b4845678664080b2b1dce5c939f60aa7079 not found: ID does not exist"
Oct 31 03:36:46.786708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.784357    5831 scope.go:122] "RemoveContainer" containerID="74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"
Oct 31 03:36:46.786708 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:36:46.784867    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769\": container with ID starting with 74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769 not found: ID does not exist" containerID="74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"
Oct 31 03:36:46.786708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.784892    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769"} err="failed to get container status \"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769\": rpc error: code = NotFound desc = could not find container \"74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769\": container with ID starting with 74defc49f42860af796836b2bcc137d3c180e8bed2deef297c01439465eba769 not found: ID does not exist"
Oct 31 03:36:46.858619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.858588    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:36:46.898088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.898036    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:36:46.902122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.902094    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-87af9c-openshift-default-85d666bc67-p799k"]
Oct 31 03:36:46.904829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.904808    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:36:46.969047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.969016    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\" (UniqueName: \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\") pod \"4210d2d9-981a-4e07-a275-4b5e887d70b4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") "
Oct 31 03:36:46.969252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.969234    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\") pod \"4210d2d9-981a-4e07-a275-4b5e887d70b4\" (UID: \"4210d2d9-981a-4e07-a275-4b5e887d70b4\") "
Oct 31 03:36:46.969666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.969635    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume" pod "4210d2d9-981a-4e07-a275-4b5e887d70b4" (UID: "4210d2d9-981a-4e07-a275-4b5e887d70b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:46.969843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.969827    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4210d2d9-981a-4e07-a275-4b5e887d70b4-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:46.971521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:46.971497    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht" pod "4210d2d9-981a-4e07-a275-4b5e887d70b4" (UID: "4210d2d9-981a-4e07-a275-4b5e887d70b4"). InnerVolumeSpecName "kube-api-access-9nmht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:47.070868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.070831    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\") pod \"d38261dc-5682-45d8-89a9-2320e5196831\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") "
Oct 31 03:36:47.071029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.070904    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\" (UniqueName: \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\") pod \"d38261dc-5682-45d8-89a9-2320e5196831\" (UID: \"d38261dc-5682-45d8-89a9-2320e5196831\") "
Oct 31 03:36:47.071300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.071266    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume" pod "d38261dc-5682-45d8-89a9-2320e5196831" (UID: "d38261dc-5682-45d8-89a9-2320e5196831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:47.071471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.071455    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38261dc-5682-45d8-89a9-2320e5196831-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:47.071603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.071473    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmht\" (UniqueName: \"kubernetes.io/projected/4210d2d9-981a-4e07-a275-4b5e887d70b4-kube-api-access-9nmht\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:47.073245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.073220    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf" pod "d38261dc-5682-45d8-89a9-2320e5196831" (UID: "d38261dc-5682-45d8-89a9-2320e5196831"). InnerVolumeSpecName "kube-api-access-rxdqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:47.172329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.172296    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdqf\" (UniqueName: \"kubernetes.io/projected/d38261dc-5682-45d8-89a9-2320e5196831-kube-api-access-rxdqf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:47.288572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.288471    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:36:47.289698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.289667    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:36:47.758335 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.758305    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"
Oct 31 03:36:47.758872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.758340    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf" event={"ID":"d38261dc-5682-45d8-89a9-2320e5196831","Type":"ContainerDied","Data":"5c1e8412618114cea46f100e928e7f4e686d1499c2ba95fec8d06070d4640680"}
Oct 31 03:36:47.758872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.758403    5831 scope.go:122] "RemoveContainer" containerID="c440897344225ebb874eff1dbb2a410591f924022f5f7ce3fc7f39349f3cf88e"
Oct 31 03:36:47.760095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.760079    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"
Oct 31 03:36:47.760202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.760165    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4" event={"ID":"4210d2d9-981a-4e07-a275-4b5e887d70b4","Type":"ContainerDied","Data":"5057960c4f348c6c8ab95222e03d11694c9f1fcc9274f20d514be94af7e42ba3"}
Oct 31 03:36:47.771658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.771627    5831 scope.go:122] "RemoveContainer" containerID="0b938a2ae2fe88a5a7da72c9e3766dd2b9b631c209b6373fb647fed037cef269"
Oct 31 03:36:47.924152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.924017    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:36:47.927726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:47.927688    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-7b7c9cb8b9-fbhj4"]
Oct 31 03:36:48.072399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.072265    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:36:48.202010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.201924    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4210d2d9-981a-4e07-a275-4b5e887d70b4" path="/var/lib/kubelet/pods/4210d2d9-981a-4e07-a275-4b5e887d70b4/volumes"
Oct 31 03:36:48.203521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.203486    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" path="/var/lib/kubelet/pods/6925218a-23f2-40b0-8762-c6adc4632afa/volumes"
Oct 31 03:36:48.206811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.206780    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-5b759cbf8c-7jcnf"]
Oct 31 03:36:48.396742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.396654    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:36:48.397469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397445    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d38261dc-5682-45d8-89a9-2320e5196831" containerName="payload-pre-processing"
Oct 31 03:36:48.397469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397469    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38261dc-5682-45d8-89a9-2320e5196831" containerName="payload-pre-processing"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397511    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397519    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397545    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397553    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397573    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4210d2d9-981a-4e07-a275-4b5e887d70b4" containerName="payload-processing"
Oct 31 03:36:48.397694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397580    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4210d2d9-981a-4e07-a275-4b5e887d70b4" containerName="payload-processing"
Oct 31 03:36:48.397999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397787    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38261dc-5682-45d8-89a9-2320e5196831" containerName="payload-pre-processing"
Oct 31 03:36:48.397999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397805    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397815    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6925218a-23f2-40b0-8762-c6adc4632afa" containerName="istio-proxy"
Oct 31 03:36:48.397999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.397825    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4210d2d9-981a-4e07-a275-4b5e887d70b4" containerName="payload-processing"
Oct 31 03:36:48.403126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.403105    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.426711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.426688    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.426872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.426849    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4zx\" (UniqueName: \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.457552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.454610    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:36:48.540444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.540401    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-18d5f6-dockercfg-pjw6v\""
Oct 31 03:36:48.540811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.540774    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4zx\" (UniqueName: \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.598591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.540936    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.598591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.579024    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-18d5f6\""
Oct 31 03:36:48.598591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.588484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.691554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.687375    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4zx\" (UniqueName: \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.692741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.692712    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:36:48.708562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.706067    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:48.708562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.707128    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:36:48.719899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.719872    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:48.868302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.868266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:48.868811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.868336    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj22\" (UniqueName: \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:48.970070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.969986    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj22\" (UniqueName: \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:48.970218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.970179    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:48.970918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:48.970899    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:49.023742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.023705    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj22\" (UniqueName: \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\") pod \"payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:49.029423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.029399    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:49.149595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.149565    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:36:49.493598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.493560    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:36:49.495663 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:49.495633    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0514f03_0b20_422a_8630_ee4cf0c696dd.slice/crio-c5bf5081171f645d690a769f41cdd9fc3c0d16259e8019df47249226c5f3f1ae WatchSource:0}: Error finding container c5bf5081171f645d690a769f41cdd9fc3c0d16259e8019df47249226c5f3f1ae: Status 404 returned error can't find the container with id c5bf5081171f645d690a769f41cdd9fc3c0d16259e8019df47249226c5f3f1ae
Oct 31 03:36:49.783921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.783738    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" event={"ID":"f0514f03-0b20-422a-8630-ee4cf0c696dd","Type":"ContainerStarted","Data":"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"}
Oct 31 03:36:49.783921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.783812    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:36:49.783921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.783834    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" event={"ID":"f0514f03-0b20-422a-8630-ee4cf0c696dd","Type":"ContainerStarted","Data":"c5bf5081171f645d690a769f41cdd9fc3c0d16259e8019df47249226c5f3f1ae"}
Oct 31 03:36:49.786043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.786009    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" event={"ID":"ce4b6b9d-87e7-4358-8a37-6233a59fbb77","Type":"ContainerStarted","Data":"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"}
Oct 31 03:36:49.786190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.786112    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:36:49.786265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.786194    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" event={"ID":"ce4b6b9d-87e7-4358-8a37-6233a59fbb77","Type":"ContainerStarted","Data":"b1321e43dac7688bf1be0e13df72bbf09e05d23e61e955b1d4694c445439d1cf"}
Oct 31 03:36:49.891655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.891596    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" podStartSLOduration=1.8915774189999999 podStartE2EDuration="1.891577419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:49.890156345 +0000 UTC m=+1759.904718054" watchObservedRunningTime="2026-10-31 03:36:49.891577419 +0000 UTC m=+1759.906139128"
Oct 31 03:36:49.999729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:49.999664    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" podStartSLOduration=1.999644268 podStartE2EDuration="1.999644268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:49.995107433 +0000 UTC m=+1760.009669143" watchObservedRunningTime="2026-10-31 03:36:49.999644268 +0000 UTC m=+1760.014205976"
Oct 31 03:36:50.184099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:50.177658    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38261dc-5682-45d8-89a9-2320e5196831" path="/var/lib/kubelet/pods/d38261dc-5682-45d8-89a9-2320e5196831/volumes"
Oct 31 03:36:50.970596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:50.969480    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:36:50.988556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:50.987897    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.033595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.021030    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2qh\" (UniqueName: \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.033948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.033921    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.044552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.041018    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5d3964\""
Oct 31 03:36:51.048554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.046803    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:36:51.064993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.064960    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5d3964-dockercfg-wx4qc\""
Oct 31 03:36:51.108803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.108196    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:36:51.141791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.141509    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.422597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.415776    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2qh\" (UniqueName: \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.423014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.422988    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.424520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.424495    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.426599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.426581    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:36:51.480172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.480135    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2qh\" (UniqueName: \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\") pod \"payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.524635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.524504    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.524804 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.524651    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhpx\" (UniqueName: \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.626101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.626066    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.626310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.626155    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhpx\" (UniqueName: \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.627000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.626973    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.647921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.647895    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:51.680328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.680290    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhpx\" (UniqueName: \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\") pod \"payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:51.737767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:51.737721    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:52.037195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.037158    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:36:52.037670 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:52.037586    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c47c73_ddc8_4218_92d9_26b83d21507b.slice/crio-db5c13b1d7c1a0f711fd1a0d563065f7c3600886b0c49fa0bf14820f4fdb01e3 WatchSource:0}: Error finding container db5c13b1d7c1a0f711fd1a0d563065f7c3600886b0c49fa0bf14820f4fdb01e3: Status 404 returned error can't find the container with id db5c13b1d7c1a0f711fd1a0d563065f7c3600886b0c49fa0bf14820f4fdb01e3
Oct 31 03:36:52.141285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.141255    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:36:52.143355 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:52.143327    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc081e01_3334_46d1_928b_6ff0a2c494fd.slice/crio-1ad2c4d0cad874774d1e978c6042a7da7f8368a7e493d0fb81d6d27822bef4fb WatchSource:0}: Error finding container 1ad2c4d0cad874774d1e978c6042a7da7f8368a7e493d0fb81d6d27822bef4fb: Status 404 returned error can't find the container with id 1ad2c4d0cad874774d1e978c6042a7da7f8368a7e493d0fb81d6d27822bef4fb
Oct 31 03:36:52.840582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.840521    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" event={"ID":"dc081e01-3334-46d1-928b-6ff0a2c494fd","Type":"ContainerStarted","Data":"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"}
Oct 31 03:36:52.840800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.840614    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:36:52.840800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.840643    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" event={"ID":"dc081e01-3334-46d1-928b-6ff0a2c494fd","Type":"ContainerStarted","Data":"1ad2c4d0cad874774d1e978c6042a7da7f8368a7e493d0fb81d6d27822bef4fb"}
Oct 31 03:36:52.842319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.842294    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" event={"ID":"23c47c73-ddc8-4218-92d9-26b83d21507b","Type":"ContainerStarted","Data":"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"}
Oct 31 03:36:52.842435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.842336    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" event={"ID":"23c47c73-ddc8-4218-92d9-26b83d21507b","Type":"ContainerStarted","Data":"db5c13b1d7c1a0f711fd1a0d563065f7c3600886b0c49fa0bf14820f4fdb01e3"}
Oct 31 03:36:52.842435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.842380    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:36:52.947968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:52.947904    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" podStartSLOduration=1.947884954 podStartE2EDuration="1.947884954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:52.945972534 +0000 UTC m=+1762.960534243" watchObservedRunningTime="2026-10-31 03:36:52.947884954 +0000 UTC m=+1762.962446663"
Oct 31 03:36:53.054179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:53.054118    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" podStartSLOduration=3.054104532 podStartE2EDuration="3.054104532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:53.050142171 +0000 UTC m=+1763.064703878" watchObservedRunningTime="2026-10-31 03:36:53.054104532 +0000 UTC m=+1763.068666239"
Oct 31 03:36:55.044763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:55.044732    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"]
Oct 31 03:36:55.090520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:55.090478    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420a558d-wv4zq"]
Oct 31 03:36:56.167141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:56.162702    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231284c4-27b1-4e64-816a-bffe8c8c8f48" path="/var/lib/kubelet/pods/231284c4-27b1-4e64-816a-bffe8c8c8f48/volumes"
Oct 31 03:36:56.244395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:56.244358    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:36:58.220666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.220038    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:36:58.245825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.237963    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.268889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.268860    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:36:58.323556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.309137    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a32cdb\""
Oct 31 03:36:58.323556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.309314    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a32cdb-dockercfg-78fjz\""
Oct 31 03:36:58.385772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.385720    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.387400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.387363    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwplz\" (UniqueName: \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.391722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.391696    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:36:58.428118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.428086    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.451851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.451760    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:36:58.506607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.506489    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwplz\" (UniqueName: \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.506954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.506931    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86cdk\" (UniqueName: \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.507150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.507135    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.507263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.507247    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.508297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.508270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.561974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.561938    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwplz\" (UniqueName: \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.580156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.579908    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:36:58.608481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.608445    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86cdk\" (UniqueName: \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.608651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.608585    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.609253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.609229    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.665640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.665573    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86cdk\" (UniqueName: \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\") pod \"payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.752904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.752870    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:58.960601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:58.960574    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:36:58.961596 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:58.961568    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b556e4a_215c_4bd6_9e2e_fc954396da0d.slice/crio-13c1d9cdb3be04f389cf6907b3cf6ad1d10f7c3b72bf264e067c8ff251451a07 WatchSource:0}: Error finding container 13c1d9cdb3be04f389cf6907b3cf6ad1d10f7c3b72bf264e067c8ff251451a07: Status 404 returned error can't find the container with id 13c1d9cdb3be04f389cf6907b3cf6ad1d10f7c3b72bf264e067c8ff251451a07
Oct 31 03:36:59.130959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.130931    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:36:59.132035 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:36:59.132011    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d023f6_cc69_4216_9c6b_8dad180323f7.slice/crio-ff8574ca1a8357ef3c3ce202671c1a861be2dfcfa6816e5e37009a4bf1cb76eb WatchSource:0}: Error finding container ff8574ca1a8357ef3c3ce202671c1a861be2dfcfa6816e5e37009a4bf1cb76eb: Status 404 returned error can't find the container with id ff8574ca1a8357ef3c3ce202671c1a861be2dfcfa6816e5e37009a4bf1cb76eb
Oct 31 03:36:59.907766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.907717    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" event={"ID":"91d023f6-cc69-4216-9c6b-8dad180323f7","Type":"ContainerStarted","Data":"79c5c90fef87af9ffda057786e56284d046fa7af60d00cae6f0910d2ade3ebb5"}
Oct 31 03:36:59.908266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.907831    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" event={"ID":"91d023f6-cc69-4216-9c6b-8dad180323f7","Type":"ContainerStarted","Data":"ff8574ca1a8357ef3c3ce202671c1a861be2dfcfa6816e5e37009a4bf1cb76eb"}
Oct 31 03:36:59.908266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.908080    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:36:59.909510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.909484    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" event={"ID":"3b556e4a-215c-4bd6-9e2e-fc954396da0d","Type":"ContainerStarted","Data":"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"}
Oct 31 03:36:59.909650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.909541    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" event={"ID":"3b556e4a-215c-4bd6-9e2e-fc954396da0d","Type":"ContainerStarted","Data":"13c1d9cdb3be04f389cf6907b3cf6ad1d10f7c3b72bf264e067c8ff251451a07"}
Oct 31 03:36:59.909650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:36:59.909581    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:37:00.013940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:00.013883    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" podStartSLOduration=2.013869562 podStartE2EDuration="2.013869562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:00.012615772 +0000 UTC m=+1770.027177480" watchObservedRunningTime="2026-10-31 03:37:00.013869562 +0000 UTC m=+1770.028431267"
Oct 31 03:37:00.147774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:00.142983    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" podStartSLOduration=2.142962019 podStartE2EDuration="2.142962019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:00.127762433 +0000 UTC m=+1770.142324143" watchObservedRunningTime="2026-10-31 03:37:00.142962019 +0000 UTC m=+1770.157523728"
Oct 31 03:37:00.799898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:00.799867    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:37:00.800079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:00.799948    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:37:03.853999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:03.853968    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:37:03.854445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:03.854043    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:37:10.924699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:10.924670    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:37:10.925185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:10.924742    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:37:12.979141 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:12.979106    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:37:14.136463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:14.136431    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:37:14.303558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:14.300607    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:37:19.136955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.136905    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" podUID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" containerName="payload-processing" containerID="cri-o://46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d" gracePeriod=25
Oct 31 03:37:19.302047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.301828    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" podUID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" containerName="payload-pre-processing" containerID="cri-o://3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee" gracePeriod=25
Oct 31 03:37:19.577569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.576365    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:37:19.584269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.584144    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\") pod \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") "
Oct 31 03:37:19.584548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.584452    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\") pod \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\" (UID: \"13d1b43c-813b-4dc6-a892-acb0930e8ff0\") "
Oct 31 03:37:19.585019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.584617    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume" pod "13d1b43c-813b-4dc6-a892-acb0930e8ff0" (UID: "13d1b43c-813b-4dc6-a892-acb0930e8ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:19.585147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.585080    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b43c-813b-4dc6-a892-acb0930e8ff0-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:19.587602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.587563    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf" pod "13d1b43c-813b-4dc6-a892-acb0930e8ff0" (UID: "13d1b43c-813b-4dc6-a892-acb0930e8ff0"). InnerVolumeSpecName "kube-api-access-2nfbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:19.778163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.777060    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/13d1b43c-813b-4dc6-a892-acb0930e8ff0-kube-api-access-2nfbf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:19.836845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.836819    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:37:19.877805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.877773    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\") pod \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") "
Oct 31 03:37:19.877971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.877951    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\" (UniqueName: \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\") pod \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\" (UID: \"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb\") "
Oct 31 03:37:19.878208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.878188    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume" pod "37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" (UID: "37201e5a-baf9-43e2-97c5-b37bf3c5e9fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:19.878494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.878478    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:19.880246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:19.880221    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4" pod "37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" (UID: "37201e5a-baf9-43e2-97c5-b37bf3c5e9fb"). InnerVolumeSpecName "kube-api-access-k5pn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:20.065079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.065029    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5pn4\" (UniqueName: \"kubernetes.io/projected/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb-kube-api-access-k5pn4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.103628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.100324    5831 generic.go:356] "Generic (PLEG): container finished" podID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" containerID="46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d" exitCode=0
Oct 31 03:37:20.103628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.100522    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" event={"ID":"13d1b43c-813b-4dc6-a892-acb0930e8ff0","Type":"ContainerDied","Data":"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"}
Oct 31 03:37:20.103628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.100617    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw" event={"ID":"13d1b43c-813b-4dc6-a892-acb0930e8ff0","Type":"ContainerDied","Data":"fe3fc545410a49602ffce3aa2d8b2702ca0f030e50c7b72424f672eac0cef645"}
Oct 31 03:37:20.103628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.100649    5831 scope.go:122] "RemoveContainer" containerID="46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"
Oct 31 03:37:20.103628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.100811    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"
Oct 31 03:37:20.111847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.110199    5831 generic.go:356] "Generic (PLEG): container finished" podID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" containerID="3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee" exitCode=0
Oct 31 03:37:20.111847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.110324    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" event={"ID":"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb","Type":"ContainerDied","Data":"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"}
Oct 31 03:37:20.111847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.110370    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx" event={"ID":"37201e5a-baf9-43e2-97c5-b37bf3c5e9fb","Type":"ContainerDied","Data":"417cf5d59778c30ce283f2e8eb75be3512a3aea817b2876b6224a47d8ca97e94"}
Oct 31 03:37:20.111847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.110440    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"
Oct 31 03:37:20.177707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.149918    5831 scope.go:122] "RemoveContainer" containerID="46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"
Oct 31 03:37:20.177707 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:20.150465    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d\": container with ID starting with 46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d not found: ID does not exist" containerID="46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"
Oct 31 03:37:20.177707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.150514    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d"} err="failed to get container status \"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d\": rpc error: code = NotFound desc = could not find container \"46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d\": container with ID starting with 46f1b6db59d7c1ac3b7089739e2dac38ba680d22be80d62d329e3d5f778bb41d not found: ID does not exist"
Oct 31 03:37:20.177707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.162590    5831 scope.go:122] "RemoveContainer" containerID="3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"
Oct 31 03:37:20.259614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.245303    5831 scope.go:122] "RemoveContainer" containerID="3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"
Oct 31 03:37:20.259806 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:20.259767    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee\": container with ID starting with 3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee not found: ID does not exist" containerID="3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"
Oct 31 03:37:20.271603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.259829    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee"} err="failed to get container status \"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee\": rpc error: code = NotFound desc = could not find container \"3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee\": container with ID starting with 3e86684f5672a366ff49037334fbf5052c51207da6ba84646abe14da0bff06ee not found: ID does not exist"
Oct 31 03:37:20.281848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.281792    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:37:20.289926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.289654    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-64c6b97d5f-l7tkw"]
Oct 31 03:37:20.477663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.477596    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:37:20.520087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.520019    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-f4c5f9f47-4cmbx"]
Oct 31 03:37:20.993993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.993960    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:37:20.994909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.994884    5831 cpu_manager.go:414] "containerMap: removing container" podUID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" containerName="payload-processing"
Oct 31 03:37:20.994909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.994908    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" containerName="payload-processing"
Oct 31 03:37:20.995128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.994928    5831 cpu_manager.go:414] "containerMap: removing container" podUID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" containerName="payload-pre-processing"
Oct 31 03:37:20.995128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.994936    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" containerName="payload-pre-processing"
Oct 31 03:37:20.995128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.995110    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" containerName="payload-pre-processing"
Oct 31 03:37:20.995284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:20.995143    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" containerName="payload-processing"
Oct 31 03:37:21.000226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.000205    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.017840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.017813    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5swb\" (UniqueName: \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.018075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.018052    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.050162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.050135    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:37:21.091718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.091687    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5d3f8a\""
Oct 31 03:37:21.091935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.091712    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5d3f8a-dockercfg-k7f96\""
Oct 31 03:37:21.120290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.120253    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5swb\" (UniqueName: \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.120587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.120515    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.121914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.121884    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.173863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.173836    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5swb\" (UniqueName: \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\") pod \"payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.316336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.316252    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:21.692044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.692007    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:37:21.692654 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:37:21.692628    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02ad148_a71b_4078_8eb3_d9bb4f2319ed.slice/crio-c04c8c1ef63e781cf8526302f98c106d1f27713fcddd15d6fe27b62ff667fd12 WatchSource:0}: Error finding container c04c8c1ef63e781cf8526302f98c106d1f27713fcddd15d6fe27b62ff667fd12: Status 404 returned error can't find the container with id c04c8c1ef63e781cf8526302f98c106d1f27713fcddd15d6fe27b62ff667fd12
Oct 31 03:37:21.729117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.729086    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:37:21.734758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.734734    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.794204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.794168    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:37:21.831294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.831248    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644kn\" (UniqueName: \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.831472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.831308    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.932468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.932429    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644kn\" (UniqueName: \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.932468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.932473    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.933385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.933352    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:21.986826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:21.986736    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-644kn\" (UniqueName: \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\") pod \"payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:22.050793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:22.050764    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:22.145817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:22.145773    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d1b43c-813b-4dc6-a892-acb0930e8ff0" path="/var/lib/kubelet/pods/13d1b43c-813b-4dc6-a892-acb0930e8ff0/volumes"
Oct 31 03:37:22.146375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:22.146354    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37201e5a-baf9-43e2-97c5-b37bf3c5e9fb" path="/var/lib/kubelet/pods/37201e5a-baf9-43e2-97c5-b37bf3c5e9fb/volumes"
Oct 31 03:37:22.152382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:22.152359    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" event={"ID":"d02ad148-a71b-4078-8eb3-d9bb4f2319ed","Type":"ContainerStarted","Data":"c04c8c1ef63e781cf8526302f98c106d1f27713fcddd15d6fe27b62ff667fd12"}
Oct 31 03:37:22.431454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:22.431425    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:37:22.432817 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:37:22.432791    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b0d556_17be_49d2_8bef_ac6e69cc36c4.slice/crio-d306c3576d782163f96b4b879c5aa666e481fa258da8b0ee1d9faf34547b4f40 WatchSource:0}: Error finding container d306c3576d782163f96b4b879c5aa666e481fa258da8b0ee1d9faf34547b4f40: Status 404 returned error can't find the container with id d306c3576d782163f96b4b879c5aa666e481fa258da8b0ee1d9faf34547b4f40
Oct 31 03:37:23.164879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:23.164841    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" event={"ID":"29b0d556-17be-49d2-8bef-ac6e69cc36c4","Type":"ContainerStarted","Data":"d306c3576d782163f96b4b879c5aa666e481fa258da8b0ee1d9faf34547b4f40"}
Oct 31 03:37:25.203780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.203737    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" event={"ID":"d02ad148-a71b-4078-8eb3-d9bb4f2319ed","Type":"ContainerStarted","Data":"fb9107f156974c5643a7e9029f4d2253cdc8e886453315f11a91f8862f712313"}
Oct 31 03:37:25.204374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.204345    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:25.205576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.205555    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" event={"ID":"29b0d556-17be-49d2-8bef-ac6e69cc36c4","Type":"ContainerStarted","Data":"c83befe83207d4bcc35bf34c4aa23a5dd88ccee338d4fef0c97727fc2de69552"}
Oct 31 03:37:25.206105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.206089    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:25.311503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.311378    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" podStartSLOduration=2.22879195 podStartE2EDuration="5.311358765s" totalImagesPullingTime="3.082566815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.308273286 +0000 UTC m=+1795.322834976" watchObservedRunningTime="2026-10-31 03:37:25.311358765 +0000 UTC m=+1795.325920474"
Oct 31 03:37:25.418009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:25.417942    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" podStartSLOduration=1.8727927 podStartE2EDuration="4.417922345s" totalImagesPullingTime="2.545129645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.41418904 +0000 UTC m=+1795.428750747" watchObservedRunningTime="2026-10-31 03:37:25.417922345 +0000 UTC m=+1795.432484056"
Oct 31 03:37:28.124244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.124212    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:37:28.129295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.129263    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.178283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.178217    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e1aad0-dockercfg-4hjzv\""
Oct 31 03:37:28.178476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.178289    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e1aad0\""
Oct 31 03:37:28.181968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.181946    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:37:28.233091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.233024    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.233286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.233268    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gp8\" (UniqueName: \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.334649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.334618    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.334828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.334768    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gp8\" (UniqueName: \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.335325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.335305    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.388229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.388157    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gp8\" (UniqueName: \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\") pod \"payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.447243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.447206    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:28.830386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.830352    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:37:28.831946 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:37:28.831916    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1760af4c_e52e_42fa_8ba0_8018e70c6a5e.slice/crio-3835360c008408b7b1a51e8f0b4c4c4903e46a1733e6bb517bbfb56a758938ac WatchSource:0}: Error finding container 3835360c008408b7b1a51e8f0b4c4c4903e46a1733e6bb517bbfb56a758938ac: Status 404 returned error can't find the container with id 3835360c008408b7b1a51e8f0b4c4c4903e46a1733e6bb517bbfb56a758938ac
Oct 31 03:37:28.887046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.887016    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:37:28.893079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.893060    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:28.942305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.942270    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sr8\" (UniqueName: \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:28.942481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.942345    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:28.944461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.944438    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:37:28.998171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.998136    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:37:28.998409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:28.998385    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" podUID="d046121f-f7f2-4d25-a8db-f615c1cf2832" containerName="maas-api" containerID="cri-o://70f2247aa9386cc42cd598c3c07f6441c14f806a28e6026cb30c4b8823a28975" gracePeriod=30
Oct 31 03:37:29.044430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.044387    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sr8\" (UniqueName: \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:29.044629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.044474    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:29.045469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.045442    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:29.102675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.102602    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sr8\" (UniqueName: \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\") pod \"payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:29.248484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.219783    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:29.323424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.323193    5831 generic.go:356] "Generic (PLEG): container finished" podID="d046121f-f7f2-4d25-a8db-f615c1cf2832" containerID="70f2247aa9386cc42cd598c3c07f6441c14f806a28e6026cb30c4b8823a28975" exitCode=0
Oct 31 03:37:29.323424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.323317    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" event={"ID":"d046121f-f7f2-4d25-a8db-f615c1cf2832","Type":"ContainerDied","Data":"70f2247aa9386cc42cd598c3c07f6441c14f806a28e6026cb30c4b8823a28975"}
Oct 31 03:37:29.330565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.330500    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" event={"ID":"1760af4c-e52e-42fa-8ba0-8018e70c6a5e","Type":"ContainerStarted","Data":"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"}
Oct 31 03:37:29.345566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.331919    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" event={"ID":"1760af4c-e52e-42fa-8ba0-8018e70c6a5e","Type":"ContainerStarted","Data":"3835360c008408b7b1a51e8f0b4c4c4903e46a1733e6bb517bbfb56a758938ac"}
Oct 31 03:37:29.346039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.346001    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:29.476554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.474132    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" podStartSLOduration=1.474109085 podStartE2EDuration="1.474109085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:29.438550715 +0000 UTC m=+1799.453112419" watchObservedRunningTime="2026-10-31 03:37:29.474109085 +0000 UTC m=+1799.488670790"
Oct 31 03:37:29.665827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.665784    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:37:29.729853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.729819    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\") pod \"d046121f-f7f2-4d25-a8db-f615c1cf2832\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") "
Oct 31 03:37:29.730016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.729997    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\" (UniqueName: \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\") pod \"d046121f-f7f2-4d25-a8db-f615c1cf2832\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") "
Oct 31 03:37:29.730107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.730038    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") pod \"d046121f-f7f2-4d25-a8db-f615c1cf2832\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") "
Oct 31 03:37:29.730107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.730089    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\") pod \"d046121f-f7f2-4d25-a8db-f615c1cf2832\" (UID: \"d046121f-f7f2-4d25-a8db-f615c1cf2832\") "
Oct 31 03:37:29.731311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.731278    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca" pod "d046121f-f7f2-4d25-a8db-f615c1cf2832" (UID: "d046121f-f7f2-4d25-a8db-f615c1cf2832"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:29.735631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.735522    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls" pod "d046121f-f7f2-4d25-a8db-f615c1cf2832" (UID: "d046121f-f7f2-4d25-a8db-f615c1cf2832"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:37:29.744256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.744236    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts" pod "d046121f-f7f2-4d25-a8db-f615c1cf2832" (UID: "d046121f-f7f2-4d25-a8db-f615c1cf2832"). InnerVolumeSpecName "kube-api-access-7hmts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:29.744367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.744233    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs" pod "d046121f-f7f2-4d25-a8db-f615c1cf2832" (UID: "d046121f-f7f2-4d25-a8db-f615c1cf2832"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:37:29.832772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.832728    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:29.832772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.832767    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmts\" (UniqueName: \"kubernetes.io/projected/d046121f-f7f2-4d25-a8db-f615c1cf2832-kube-api-access-7hmts\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:29.832992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.832783    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d046121f-f7f2-4d25-a8db-f615c1cf2832-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:29.832992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:29.832796    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d046121f-f7f2-4d25-a8db-f615c1cf2832-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:30.011195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.005939    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:37:30.343176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.343140    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"
Oct 31 03:37:30.343696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.343186    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw" event={"ID":"d046121f-f7f2-4d25-a8db-f615c1cf2832","Type":"ContainerDied","Data":"fb349a51873bb98c1a9155bc96c86b59dfddeada1de393931892ddf5a8567d82"}
Oct 31 03:37:30.343696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.343246    5831 scope.go:122] "RemoveContainer" containerID="70f2247aa9386cc42cd598c3c07f6441c14f806a28e6026cb30c4b8823a28975"
Oct 31 03:37:30.345218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.345188    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" event={"ID":"8f269c40-4e5e-447c-a794-f24aceb3762e","Type":"ContainerStarted","Data":"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"}
Oct 31 03:37:30.345360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.345236    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" event={"ID":"8f269c40-4e5e-447c-a794-f24aceb3762e","Type":"ContainerStarted","Data":"5ff6315d2d174e97ba59d64ac7097dc6a4eddbce734bbfef510721ed803df3e0"}
Oct 31 03:37:30.497453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.497366    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:37:30.501075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.501046    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b3a29ec-7997f7f797-lpqhw"]
Oct 31 03:37:30.600849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.600783    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" podStartSLOduration=2.600764498 podStartE2EDuration="2.600764498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:30.600211318 +0000 UTC m=+1800.614773027" watchObservedRunningTime="2026-10-31 03:37:30.600764498 +0000 UTC m=+1800.615326207"
Oct 31 03:37:30.657964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.657930    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:37:30.658703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.658679    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d046121f-f7f2-4d25-a8db-f615c1cf2832" containerName="maas-api"
Oct 31 03:37:30.658703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.658698    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d046121f-f7f2-4d25-a8db-f615c1cf2832" containerName="maas-api"
Oct 31 03:37:30.658914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.658849    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d046121f-f7f2-4d25-a8db-f615c1cf2832" containerName="maas-api"
Oct 31 03:37:30.664115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.664087    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.718083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.718042    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:37:30.811296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.809433    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.811296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.809567    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.811860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.811819    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-18d5f6-serving-cert\""
Oct 31 03:37:30.817514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.813070    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.817514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.813146    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9mj\" (UniqueName: \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.914253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.914203    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.914441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.914356    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.914441 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:30.914380    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-serving-cert: secret "maas-api-e2e-ipp-b-18d5f6-serving-cert" not found
Oct 31 03:37:30.914441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.914407    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9mj\" (UniqueName: \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.914623 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:30.914461    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls podName:fde8be45-364e-4575-8b04-3dc768de65e7 nodeName:}" failed. No retries permitted until 2026-10-31 03:37:31.414436341 +0000 UTC m=+1801.428998032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls") pod "maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" (UID: "fde8be45-364e-4575-8b04-3dc768de65e7") : secret "maas-api-e2e-ipp-b-18d5f6-serving-cert" not found
Oct 31 03:37:30.914623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.914571    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.915145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.915122    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.917431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.917363    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:30.986451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:30.986414    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9mj\" (UniqueName: \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:31.363064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.363023    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:37:31.363624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.363271    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy" containerID="cri-o://22a858bde9571f0694530643bce3b8e2538870594c5b89f295cfce675d1f1253" gracePeriod=30
Oct 31 03:37:31.397609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.397572    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:37:31.398013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.397962    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy" containerID="cri-o://0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129" gracePeriod=30
Oct 31 03:37:31.421056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.421029    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:31.421428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.421400    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:37:31.422644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.422616    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy" containerID="cri-o://6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012" gracePeriod=30
Oct 31 03:37:31.429599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.429571    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:31.434557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.434276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:31.589850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:31.589804    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:32.089896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.089774    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:37:32.090784 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:37:32.090753    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde8be45_364e_4575_8b04_3dc768de65e7.slice/crio-eebb8acd8f74cc4f50192771bdcfb3c5098619594ae3d64f82d7ebaf5feb7869 WatchSource:0}: Error finding container eebb8acd8f74cc4f50192771bdcfb3c5098619594ae3d64f82d7ebaf5feb7869: Status 404 returned error can't find the container with id eebb8acd8f74cc4f50192771bdcfb3c5098619594ae3d64f82d7ebaf5feb7869
Oct 31 03:37:32.139411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.139369    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d046121f-f7f2-4d25-a8db-f615c1cf2832" path="/var/lib/kubelet/pods/d046121f-f7f2-4d25-a8db-f615c1cf2832/volumes"
Oct 31 03:37:32.428612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.428574    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" event={"ID":"fde8be45-364e-4575-8b04-3dc768de65e7","Type":"ContainerStarted","Data":"eebb8acd8f74cc4f50192771bdcfb3c5098619594ae3d64f82d7ebaf5feb7869"}
Oct 31 03:37:32.732768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.732686    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:37:32.738512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.738492    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:32.861892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:32.861856    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-329d79-openshift-default-dockercfg-dwlvm\""
Oct 31 03:37:33.081307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.019139    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:37:33.091767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.091456    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.092094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.092072    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.092479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.092454    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.092793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.092773    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.092976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.092962    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.093228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.093211    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqf\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.094461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.094440    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.094696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.094680    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.094887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.094869    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196429    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196514    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196559    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196576    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196612    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqf\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196680    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196733    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196812    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.196979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.196874    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.197139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.197030    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.197139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.197090    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.197348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.197327    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.197401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.197371    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.197649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.197619    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.199298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.199278    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.199785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.199763    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.254601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.254569    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.255887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.255847    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqf\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\") pod \"e2e-api-a-329d79-openshift-default-575867c7-zmd79\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") " pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.361965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.361854    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:33.439353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.438652    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" event={"ID":"fde8be45-364e-4575-8b04-3dc768de65e7","Type":"ContainerStarted","Data":"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"}
Oct 31 03:37:33.439353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.438728    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:33.547895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.547836    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" podStartSLOduration=2.39789255 podStartE2EDuration="3.547816915s" totalImagesPullingTime="1.149924365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:33.544816754 +0000 UTC m=+1803.559378469" watchObservedRunningTime="2026-10-31 03:37:33.547816915 +0000 UTC m=+1803.562378624"
Oct 31 03:37:33.874115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.874078    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:37:33.874625 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:37:33.874595    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5453eab_bd30_4ec8_93d9_5f8660860e42.slice/crio-c4da920b43baf934ae8bead9fd184a723c96a3349c2f67b907ef80f864ec2458 WatchSource:0}: Error finding container c4da920b43baf934ae8bead9fd184a723c96a3349c2f67b907ef80f864ec2458: Status 404 returned error can't find the container with id c4da920b43baf934ae8bead9fd184a723c96a3349c2f67b907ef80f864ec2458
Oct 31 03:37:33.877387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.877352    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:33.877481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.877428    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:33.877481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:33.877475    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:34.184624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:34.184588    5831 scope.go:122] "RemoveContainer" containerID="e3e0a4f1499c9dd735def5f29872c951af0cd25698b4e220ee27b40f3b359b9e"
Oct 31 03:37:34.207866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:34.199941    5831 scope.go:122] "RemoveContainer" containerID="7bba6a535cd773a04fb8f7bb74fd161e295ae667e98806af52dfe1b3eb91bac5"
Oct 31 03:37:34.451611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:34.451578    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerStarted","Data":"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"}
Oct 31 03:37:34.452040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:34.451658    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerStarted","Data":"c4da920b43baf934ae8bead9fd184a723c96a3349c2f67b907ef80f864ec2458"}
Oct 31 03:37:34.573996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:34.563682    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podStartSLOduration=2.563665118 podStartE2EDuration="2.563665118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:34.558306567 +0000 UTC m=+1804.572868274" watchObservedRunningTime="2026-10-31 03:37:34.563665118 +0000 UTC m=+1804.578226827"
Oct 31 03:37:35.362780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:35.362743    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:35.368318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:35.368288    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:35.469857 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:35.466811    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:35.469857 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:35.468412    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:37:36.477380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.477350    5831 generic.go:356] "Generic (PLEG): container finished" podID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerID="22a858bde9571f0694530643bce3b8e2538870594c5b89f295cfce675d1f1253" exitCode=0
Oct 31 03:37:36.477805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.477429    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" event={"ID":"b3aaa60b-f502-452f-b96c-354bec49fab7","Type":"ContainerDied","Data":"22a858bde9571f0694530643bce3b8e2538870594c5b89f295cfce675d1f1253"}
Oct 31 03:37:36.793930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.793894    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:37:36.849685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.849649    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:37:36.895386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895356    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.895610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895393    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.895610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895444    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895475    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.895610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895513    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895607    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895670    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895711    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895736    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895746    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.895913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895902    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.896221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.896221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895993    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.896221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895959    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.896221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896073    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.896221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896124    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896148    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896176    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896326    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\") pod \"02690735-c044-4790-be99-54e672e02286\" (UID: \"02690735-c044-4790-be99-54e672e02286\") "
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896353    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.895945    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896428    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896445    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.896551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.896493    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\") pod \"8c06fc31-684e-44e6-ab64-0f684831dc54\" (UID: \"8c06fc31-684e-44e6-ab64-0f684831dc54\") "
Oct 31 03:37:36.897327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897299    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02690735-c044-4790-be99-54e672e02286-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897334    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897349    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06fc31-684e-44e6-ab64-0f684831dc54-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897365    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897379    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897393    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.897420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.897407    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.898730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.898674    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.898920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.898896    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.899124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.899082    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.899208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.899182    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.899302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.899281    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.899406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.899382    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.900154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.900118    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "kube-api-access-9ppdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.901214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.901100    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.901214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.901181    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9" pod "02690735-c044-4790-be99-54e672e02286" (UID: "02690735-c044-4790-be99-54e672e02286"). InnerVolumeSpecName "kube-api-access-27jl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.901684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.901661    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.904951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.904902    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token" pod "8c06fc31-684e-44e6-ab64-0f684831dc54" (UID: "8c06fc31-684e-44e6-ab64-0f684831dc54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.998614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998581    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998614    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jl9\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-kube-api-access-27jl9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998629    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02690735-c044-4790-be99-54e672e02286-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998641    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02690735-c044-4790-be99-54e672e02286-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998655    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998668    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998680    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998693    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998705    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02690735-c044-4790-be99-54e672e02286-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998729    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06fc31-684e-44e6-ab64-0f684831dc54-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.998946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:36.998742    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppdg\" (UniqueName: \"kubernetes.io/projected/8c06fc31-684e-44e6-ab64-0f684831dc54-kube-api-access-9ppdg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.077392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.077363    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:37:37.099788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.099761    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.099931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.099838    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.099931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.099893    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.099928    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100053    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100118    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100132    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:37.100265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100212    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100257    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:37.100381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100328    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:37.100381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100345    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\") pod \"b3aaa60b-f502-452f-b96c-354bec49fab7\" (UID: \"b3aaa60b-f502-452f-b96c-354bec49fab7\") "
Oct 31 03:37:37.100567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100401    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:37.100567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.100416    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:37.101208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.101181    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.101345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.101212    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.101345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.101228    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aaa60b-f502-452f-b96c-354bec49fab7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.101345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.101242    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.101345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.101256    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.102247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.102221    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:37.102604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.102581    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "kube-api-access-szr2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:37.102683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.102624    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:37.102720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.102691    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token" pod "b3aaa60b-f502-452f-b96c-354bec49fab7" (UID: "b3aaa60b-f502-452f-b96c-354bec49fab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:37.202270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.202236    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.202270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.202269    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szr2p\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-kube-api-access-szr2p\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.202584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.202282    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.202584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.202291    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aaa60b-f502-452f-b96c-354bec49fab7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:37.227378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.227296    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:37:37.228214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.228193    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:37:37.488304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.488216    5831 generic.go:356] "Generic (PLEG): container finished" podID="02690735-c044-4790-be99-54e672e02286" containerID="0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129" exitCode=0
Oct 31 03:37:37.488304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.488281    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"
Oct 31 03:37:37.488880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.488274    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerDied","Data":"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"}
Oct 31 03:37:37.488880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.488408    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn" event={"ID":"02690735-c044-4790-be99-54e672e02286","Type":"ContainerDied","Data":"8fca904427705a644dfc0546cb4a1771bb17dd7df84c5ad7a614ce3356b113f2"}
Oct 31 03:37:37.488880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.488450    5831 scope.go:122] "RemoveContainer" containerID="0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"
Oct 31 03:37:37.491267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.491242    5831 generic.go:356] "Generic (PLEG): container finished" podID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerID="6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012" exitCode=0
Oct 31 03:37:37.491515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.491285    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"
Oct 31 03:37:37.491640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.491315    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" event={"ID":"8c06fc31-684e-44e6-ab64-0f684831dc54","Type":"ContainerDied","Data":"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"}
Oct 31 03:37:37.491640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.491612    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv" event={"ID":"8c06fc31-684e-44e6-ab64-0f684831dc54","Type":"ContainerDied","Data":"734fe32a5ec07439de4f82e078870a2c2a5c3c79047cadad00205594ed97ca45"}
Oct 31 03:37:37.496833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.496807    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"
Oct 31 03:37:37.498844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.498808    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6" event={"ID":"b3aaa60b-f502-452f-b96c-354bec49fab7","Type":"ContainerDied","Data":"0d689447573c9cf9b4600058160deb95ed60f611bdb4b39767e2f5426323ab86"}
Oct 31 03:37:37.517614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.514667    5831 scope.go:122] "RemoveContainer" containerID="82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"
Oct 31 03:37:37.529789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.529769    5831 scope.go:122] "RemoveContainer" containerID="0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"
Oct 31 03:37:37.530120 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:37.530098    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129\": container with ID starting with 0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129 not found: ID does not exist" containerID="0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"
Oct 31 03:37:37.530161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.530128    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129"} err="failed to get container status \"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129\": rpc error: code = NotFound desc = could not find container \"0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129\": container with ID starting with 0e256a7079756514e325624ed8226b8eb541e2acdaf7874bfe32cb7bed6cf129 not found: ID does not exist"
Oct 31 03:37:37.530161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.530145    5831 scope.go:122] "RemoveContainer" containerID="82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"
Oct 31 03:37:37.530399 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:37.530383    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e\": container with ID starting with 82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e not found: ID does not exist" containerID="82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"
Oct 31 03:37:37.530444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.530405    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e"} err="failed to get container status \"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e\": rpc error: code = NotFound desc = could not find container \"82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e\": container with ID starting with 82c2492ef4f4fa912901cda021cf24e72038d392f2d2d527ce561ac17dfb952e not found: ID does not exist"
Oct 31 03:37:37.530444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.530418    5831 scope.go:122] "RemoveContainer" containerID="6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"
Oct 31 03:37:37.541103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.540941    5831 scope.go:122] "RemoveContainer" containerID="6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"
Oct 31 03:37:37.541253 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:37.541229    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012\": container with ID starting with 6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012 not found: ID does not exist" containerID="6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"
Oct 31 03:37:37.541303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.541263    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012"} err="failed to get container status \"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012\": rpc error: code = NotFound desc = could not find container \"6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012\": container with ID starting with 6d90d36fc1fd498a3531336fcf8f51627dad2eca420c20e2038f05624e673012 not found: ID does not exist"
Oct 31 03:37:37.541303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.541279    5831 scope.go:122] "RemoveContainer" containerID="22a858bde9571f0694530643bce3b8e2538870594c5b89f295cfce675d1f1253"
Oct 31 03:37:37.642892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.642860    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:37:37.648246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.648218    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bdspv"]
Oct 31 03:37:37.759591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.759294    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:37:37.763007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.762983    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-bd9w6"]
Oct 31 03:37:37.853343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.853290    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:37:37.857178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:37.857151    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-wgzpn"]
Oct 31 03:37:38.068989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.068909    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:37:38.068989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.068949    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:37:38.078692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.078669    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:37:38.079112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.079077    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:37:38.082793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.082776    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:37:38.083207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.083191    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:37:38.138466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.138436    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02690735-c044-4790-be99-54e672e02286" path="/var/lib/kubelet/pods/02690735-c044-4790-be99-54e672e02286/volumes"
Oct 31 03:37:38.138967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.138952    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" path="/var/lib/kubelet/pods/8c06fc31-684e-44e6-ab64-0f684831dc54/volumes"
Oct 31 03:37:38.139382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:38.139369    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" path="/var/lib/kubelet/pods/b3aaa60b-f502-452f-b96c-354bec49fab7/volumes"
Oct 31 03:37:39.458682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:39.458649    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:37:41.423564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:41.423496    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:37:42.431905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:42.431879    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:37:42.979928 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:37:42.979891    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:37:46.072753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:46.072716    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:37:57.869683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.869637    5831 generic.go:356] "Generic (PLEG): container finished" podID="96154984-ffc5-48e8-a744-6032cddc2a00" containerID="bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45" exitCode=0
Oct 31 03:37:57.870135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.869712    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerDied","Data":"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"}
Oct 31 03:37:57.870180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.870159    5831 scope.go:122] "RemoveContainer" containerID="bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"
Oct 31 03:37:57.871673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.871650    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:57.871738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.871719    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:57.871775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:57.871760    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:37:58.903560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:58.902110    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerStarted","Data":"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"}
Oct 31 03:37:59.241515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:59.241477    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:37:59.241774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:59.241712    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:37:59.241861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:37:59.241777    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:38:00.242257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:00.242223    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:38:00.242708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:00.242275    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:38:01.241969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:01.241929    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:38:01.242164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:01.241995    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:38:02.242098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:02.242059    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:38:02.242502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:02.242132    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:38:03.241852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:03.241818    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 03:38:03.242038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:03.241864    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 03:38:04.246167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:04.246136    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:38:04.246682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:04.246562    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:38:04.247201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:04.247185    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:38:12.979013 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:38:12.978965    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:38:13.489601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:13.488059    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:38:18.945613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.945578    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:38:18.946443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946420    5831 cpu_manager.go:414] "containerMap: removing container" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946443    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946472    5831 cpu_manager.go:414] "containerMap: removing container" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946481    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946489    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946494    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946522    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946542    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946663    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3aaa60b-f502-452f-b96c-354bec49fab7" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946678    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946686    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="02690735-c044-4790-be99-54e672e02286" containerName="istio-proxy"
Oct 31 03:38:18.946691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.946693    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c06fc31-684e-44e6-ab64-0f684831dc54" containerName="istio-proxy"
Oct 31 03:38:18.950717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:18.950696    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.019571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.018861    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:38:19.046884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.046851    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8b4030\""
Oct 31 03:38:19.047102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.046949    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8b4030-dockercfg-4sg5p\""
Oct 31 03:38:19.066838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.066810    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.067002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.066967    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkr5q\" (UniqueName: \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.168910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.168864    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkr5q\" (UniqueName: \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.169183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.169158    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.169960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.169938    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.223720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.223643    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkr5q\" (UniqueName: \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\") pod \"payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.266178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.266146    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:19.733814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:19.733358    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:38:20.197161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:20.197123    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" event={"ID":"3f7eccd5-1425-4a0e-bf63-f87d50328670","Type":"ContainerStarted","Data":"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"}
Oct 31 03:38:20.197707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:20.197175    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" event={"ID":"3f7eccd5-1425-4a0e-bf63-f87d50328670","Type":"ContainerStarted","Data":"30a2c4858eb04e602dcc0928de222c8ecdce093648a5a54ad8c79a2810d1f31c"}
Oct 31 03:38:20.197707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:20.197205    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:20.306693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:20.306633    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" podStartSLOduration=2.306615906 podStartE2EDuration="2.306615906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:20.303435077 +0000 UTC m=+1850.317996786" watchObservedRunningTime="2026-10-31 03:38:20.306615906 +0000 UTC m=+1850.321177614"
Oct 31 03:38:24.045231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:24.045196    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"]
Oct 31 03:38:24.092051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:24.092011    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52275d80-qjmqp"]
Oct 31 03:38:24.140079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:24.140040    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090fedfe-0367-4015-aab1-5732719843d4" path="/var/lib/kubelet/pods/090fedfe-0367-4015-aab1-5732719843d4/volumes"
Oct 31 03:38:25.242791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.242749    5831 generic.go:356] "Generic (PLEG): container finished" podID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerID="aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7" exitCode=0
Oct 31 03:38:25.243290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.242821    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerDied","Data":"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"}
Oct 31 03:38:25.243348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.243312    5831 scope.go:122] "RemoveContainer" containerID="aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"
Oct 31 03:38:25.245099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.245072    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:38:25.245183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.245138    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:38:25.245183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:25.245171    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:38:26.252722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:26.252668    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerStarted","Data":"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"}
Oct 31 03:38:26.584995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:26.584905    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:38:26.596695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:26.596654    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:26.597018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:26.596987    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:27.582955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:27.582917    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:27.583650 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:27.583615    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:28.589557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:28.582939    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:28.589557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:28.582999    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:29.586556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:29.582996    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:29.586556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:29.583059    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:30.583557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:30.582993    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:30.583557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:30.583073    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:30.724312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:30.724276    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:38:31.209034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:31.209004    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:38:31.593930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:31.593838    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 03:38:31.594553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:31.594492    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 03:38:32.587393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:32.587363    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:38:32.588674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:32.588653    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:38:34.452634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:34.452605    5831 scope.go:122] "RemoveContainer" containerID="b062ab21068c5d04b23b1d638f28a8dc4d468b4faf0e5a3d4ba725890be7bb4f"
Oct 31 03:38:42.979357 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:38:42.979311    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:38:51.970982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:51.970940    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:38:52.096689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:52.096640    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:38:56.972069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:56.972035    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" podUID="f0514f03-0b20-422a-8630-ee4cf0c696dd" containerName="payload-processing" containerID="cri-o://c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2" gracePeriod=25
Oct 31 03:38:57.096989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.096947    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" podUID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" containerName="payload-pre-processing" containerID="cri-o://eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17" gracePeriod=25
Oct 31 03:38:57.274617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.274586    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:38:57.351640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.351605    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\" (UniqueName: \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\") pod \"f0514f03-0b20-422a-8630-ee4cf0c696dd\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") "
Oct 31 03:38:57.351852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.351701    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\") pod \"f0514f03-0b20-422a-8630-ee4cf0c696dd\" (UID: \"f0514f03-0b20-422a-8630-ee4cf0c696dd\") "
Oct 31 03:38:57.352505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.352365    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume" pod "f0514f03-0b20-422a-8630-ee4cf0c696dd" (UID: "f0514f03-0b20-422a-8630-ee4cf0c696dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:57.354221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.354199    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22" pod "f0514f03-0b20-422a-8630-ee4cf0c696dd" (UID: "f0514f03-0b20-422a-8630-ee4cf0c696dd"). InnerVolumeSpecName "kube-api-access-svj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:57.382344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.382319    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:38:57.453782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.453750    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\" (UniqueName: \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\") pod \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") "
Oct 31 03:38:57.453945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.453809    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\") pod \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\" (UID: \"ce4b6b9d-87e7-4358-8a37-6233a59fbb77\") "
Oct 31 03:38:57.454205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.454177    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume" pod "ce4b6b9d-87e7-4358-8a37-6233a59fbb77" (UID: "ce4b6b9d-87e7-4358-8a37-6233a59fbb77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:57.454273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.454252    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0514f03-0b20-422a-8630-ee4cf0c696dd-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:57.454273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.454265    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svj22\" (UniqueName: \"kubernetes.io/projected/f0514f03-0b20-422a-8630-ee4cf0c696dd-kube-api-access-svj22\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:57.456096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.456073    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx" pod "ce4b6b9d-87e7-4358-8a37-6233a59fbb77" (UID: "ce4b6b9d-87e7-4358-8a37-6233a59fbb77"). InnerVolumeSpecName "kube-api-access-7x4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:57.555982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.555897    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4zx\" (UniqueName: \"kubernetes.io/projected/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-kube-api-access-7x4zx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:57.555982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.555928    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4b6b9d-87e7-4358-8a37-6233a59fbb77-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:57.835892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.835806    5831 generic.go:356] "Generic (PLEG): container finished" podID="f0514f03-0b20-422a-8630-ee4cf0c696dd" containerID="c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2" exitCode=0
Oct 31 03:38:57.835892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.835862    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"
Oct 31 03:38:57.835892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.835884    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" event={"ID":"f0514f03-0b20-422a-8630-ee4cf0c696dd","Type":"ContainerDied","Data":"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"}
Oct 31 03:38:57.836212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.835934    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn" event={"ID":"f0514f03-0b20-422a-8630-ee4cf0c696dd","Type":"ContainerDied","Data":"c5bf5081171f645d690a769f41cdd9fc3c0d16259e8019df47249226c5f3f1ae"}
Oct 31 03:38:57.836212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.835967    5831 scope.go:122] "RemoveContainer" containerID="c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"
Oct 31 03:38:57.837800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.837769    5831 generic.go:356] "Generic (PLEG): container finished" podID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" containerID="eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17" exitCode=0
Oct 31 03:38:57.837929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.837801    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"
Oct 31 03:38:57.837929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.837852    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" event={"ID":"ce4b6b9d-87e7-4358-8a37-6233a59fbb77","Type":"ContainerDied","Data":"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"}
Oct 31 03:38:57.837929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.837908    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh" event={"ID":"ce4b6b9d-87e7-4358-8a37-6233a59fbb77","Type":"ContainerDied","Data":"b1321e43dac7688bf1be0e13df72bbf09e05d23e61e955b1d4694c445439d1cf"}
Oct 31 03:38:57.848380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.848358    5831 scope.go:122] "RemoveContainer" containerID="c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"
Oct 31 03:38:57.848740 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:38:57.848718    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2\": container with ID starting with c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2 not found: ID does not exist" containerID="c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"
Oct 31 03:38:57.848812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.848750    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2"} err="failed to get container status \"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2\": rpc error: code = NotFound desc = could not find container \"c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2\": container with ID starting with c4067081aaf0975346d169a81df903682e734c47c4fc65304a7d94b9e10e59c2 not found: ID does not exist"
Oct 31 03:38:57.848812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.848767    5831 scope.go:122] "RemoveContainer" containerID="eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"
Oct 31 03:38:57.858963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.858947    5831 scope.go:122] "RemoveContainer" containerID="eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"
Oct 31 03:38:57.859275 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:38:57.859251    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17\": container with ID starting with eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17 not found: ID does not exist" containerID="eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"
Oct 31 03:38:57.859352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.859285    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17"} err="failed to get container status \"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17\": rpc error: code = NotFound desc = could not find container \"eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17\": container with ID starting with eab862a5f1a28100b666ab553662c20bcfdbe94eb8a4ccf9fe52ff631d21de17 not found: ID does not exist"
Oct 31 03:38:57.991704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.991612    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:38:57.997334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:57.997308    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-6fdff4bcf-hmbfh"]
Oct 31 03:38:58.093928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:58.093838    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:38:58.099595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:58.099562    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-cc56d555-bspdn"]
Oct 31 03:38:58.137655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:58.137629    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" path="/var/lib/kubelet/pods/ce4b6b9d-87e7-4358-8a37-6233a59fbb77/volumes"
Oct 31 03:38:58.138111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:38:58.138096    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0514f03-0b20-422a-8630-ee4cf0c696dd" path="/var/lib/kubelet/pods/f0514f03-0b20-422a-8630-ee4cf0c696dd/volumes"
Oct 31 03:39:04.112794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:04.112733    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:39:10.249046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:10.249005    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:39:10.359854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:10.359580    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:39:12.992592 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:12.986105    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:39:13.653779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:13.653707    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:39:13.653995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:13.653796    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:39:15.249439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.249383    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" podUID="dc081e01-3334-46d1-928b-6ff0a2c494fd" containerName="payload-processing" containerID="cri-o://5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a" gracePeriod=25
Oct 31 03:39:15.360816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.360766    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" podUID="23c47c73-ddc8-4218-92d9-26b83d21507b" containerName="payload-pre-processing" containerID="cri-o://4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d" gracePeriod=25
Oct 31 03:39:15.534078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.534034    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:39:15.534585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.534325    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy" containerID="cri-o://b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb" gracePeriod=30
Oct 31 03:39:15.574861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.574793    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:39:15.575233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.575195    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" containerID="cri-o://c7e19798a61c179c4ac40c77dc797645e923109698d14c0427700753818b6c70" gracePeriod=30
Oct 31 03:39:15.666824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.666720    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.767166    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\") pod \"dc081e01-3334-46d1-928b-6ff0a2c494fd\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") "
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.767331    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\" (UniqueName: \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\") pod \"dc081e01-3334-46d1-928b-6ff0a2c494fd\" (UID: \"dc081e01-3334-46d1-928b-6ff0a2c494fd\") "
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.767543    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume" pod "dc081e01-3334-46d1-928b-6ff0a2c494fd" (UID: "dc081e01-3334-46d1-928b-6ff0a2c494fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.767860    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc081e01-3334-46d1-928b-6ff0a2c494fd-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.770351    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx" pod "dc081e01-3334-46d1-928b-6ff0a2c494fd" (UID: "dc081e01-3334-46d1-928b-6ff0a2c494fd"). InnerVolumeSpecName "kube-api-access-vmhpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.791157    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.869496    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\") pod \"23c47c73-ddc8-4218-92d9-26b83d21507b\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") "
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.869606    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\" (UniqueName: \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\") pod \"23c47c73-ddc8-4218-92d9-26b83d21507b\" (UID: \"23c47c73-ddc8-4218-92d9-26b83d21507b\") "
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.870142    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhpx\" (UniqueName: \"kubernetes.io/projected/dc081e01-3334-46d1-928b-6ff0a2c494fd-kube-api-access-vmhpx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.870981    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume" pod "23c47c73-ddc8-4218-92d9-26b83d21507b" (UID: "23c47c73-ddc8-4218-92d9-26b83d21507b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.873332    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh" pod "23c47c73-ddc8-4218-92d9-26b83d21507b" (UID: "23c47c73-ddc8-4218-92d9-26b83d21507b"). InnerVolumeSpecName "kube-api-access-xz2qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.971970    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c47c73-ddc8-4218-92d9-26b83d21507b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:15.971997    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2qh\" (UniqueName: \"kubernetes.io/projected/23c47c73-ddc8-4218-92d9-26b83d21507b-kube-api-access-xz2qh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.002130    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.086731    5831 generic.go:356] "Generic (PLEG): container finished" podID="23c47c73-ddc8-4218-92d9-26b83d21507b" containerID="4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d" exitCode=0
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.086778    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.086815    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" event={"ID":"23c47c73-ddc8-4218-92d9-26b83d21507b","Type":"ContainerDied","Data":"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"}
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.086863    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8" event={"ID":"23c47c73-ddc8-4218-92d9-26b83d21507b","Type":"ContainerDied","Data":"db5c13b1d7c1a0f711fd1a0d563065f7c3600886b0c49fa0bf14820f4fdb01e3"}
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.086904    5831 scope.go:122] "RemoveContainer" containerID="4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.088929    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc081e01-3334-46d1-928b-6ff0a2c494fd" containerID="5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a" exitCode=0
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.088975    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" event={"ID":"dc081e01-3334-46d1-928b-6ff0a2c494fd","Type":"ContainerDied","Data":"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"}
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.089005    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.089137    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh" event={"ID":"dc081e01-3334-46d1-928b-6ff0a2c494fd","Type":"ContainerDied","Data":"1ad2c4d0cad874774d1e978c6042a7da7f8368a7e493d0fb81d6d27822bef4fb"}
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.103829    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.107034    5831 scope.go:122] "RemoveContainer" containerID="4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:16.107412    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d\": container with ID starting with 4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d not found: ID does not exist" containerID="4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.107451    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d"} err="failed to get container status \"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d\": rpc error: code = NotFound desc = could not find container \"4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d\": container with ID starting with 4b021e2689e729abe802885ca593b26d162a3d6bda7dfb9df732cb62824a3c5d not found: ID does not exist"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.107479    5831 scope.go:122] "RemoveContainer" containerID="5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.129969    5831 scope.go:122] "RemoveContainer" containerID="5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:16.130282    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a\": container with ID starting with 5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a not found: ID does not exist" containerID="5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.130307    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a"} err="failed to get container status \"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a\": rpc error: code = NotFound desc = could not find container \"5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a\": container with ID starting with 5af5984915c66676aa4a64f8d9ee83a2fc79ac7f86d4f62b7e06dca969cad56a not found: ID does not exist"
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.241441    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.254043    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d3964-7c75f7b9b7-bp9q8"]
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.346748    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:39:16.513932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:16.351634    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d3964-5566fb7949-kpzrh"]
Oct 31 03:39:17.100386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.100303    5831 generic.go:356] "Generic (PLEG): container finished" podID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerID="61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8" exitCode=0
Oct 31 03:39:17.100579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.100381    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerDied","Data":"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"}
Oct 31 03:39:17.100930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.100905    5831 scope.go:122] "RemoveContainer" containerID="61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"
Oct 31 03:39:17.102444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.102415    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.102578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.102485    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.102578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.102521    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.106363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.106326    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerID="0c632bfc639b1a5dec6b455e3f3a1c6209a0bc465f30f10890df23bcfd30e331" exitCode=0
Oct 31 03:39:17.106496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.106405    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerDied","Data":"0c632bfc639b1a5dec6b455e3f3a1c6209a0bc465f30f10890df23bcfd30e331"}
Oct 31 03:39:17.106865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.106846    5831 scope.go:122] "RemoveContainer" containerID="0c632bfc639b1a5dec6b455e3f3a1c6209a0bc465f30f10890df23bcfd30e331"
Oct 31 03:39:17.108588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.108555    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.108684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.108666    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.108734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.108720    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:39:17.969275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.968193    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rpg88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:17.969275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:17.968261    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:18.152148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.152120    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c47c73-ddc8-4218-92d9-26b83d21507b" path="/var/lib/kubelet/pods/23c47c73-ddc8-4218-92d9-26b83d21507b/volumes"
Oct 31 03:39:18.152931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.152913    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc081e01-3334-46d1-928b-6ff0a2c494fd" path="/var/lib/kubelet/pods/dc081e01-3334-46d1-928b-6ff0a2c494fd/volumes"
Oct 31 03:39:18.153375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.153355    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerStarted","Data":"46c579f58f9783ff5ef6464e7864f53853ac35094c0e3019a26f21785770865f"}
Oct 31 03:39:18.154287 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.154259    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerStarted","Data":"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"}
Oct 31 03:39:18.363143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.363052    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:39:18.364166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.364139    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:18.364303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.364188    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:18.999578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.998971    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:39:18.999578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.999136    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 03:39:18.999578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:18.999186    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 03:39:19.363111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:19.363028    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:19.363111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:19.363084    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:19.997555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:19.994948    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 03:39:19.997555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:19.995014    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 03:39:20.363006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:20.362928    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:20.363452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:20.363367    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:20.466776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:20.466737    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:39:21.029570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.010922    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 03:39:21.029570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.011012    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 03:39:21.038870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.023690    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" podUID="91d023f6-cc69-4216-9c6b-8dad180323f7" containerName="payload-processing" containerID="cri-o://79c5c90fef87af9ffda057786e56284d046fa7af60d00cae6f0910d2ade3ebb5" gracePeriod=25
Oct 31 03:39:21.134582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.133848    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" podUID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" containerName="payload-pre-processing" containerID="cri-o://f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce" gracePeriod=25
Oct 31 03:39:21.162507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.162478    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:39:21.226554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.223555    5831 generic.go:356] "Generic (PLEG): container finished" podID="91d023f6-cc69-4216-9c6b-8dad180323f7" containerID="79c5c90fef87af9ffda057786e56284d046fa7af60d00cae6f0910d2ade3ebb5" exitCode=0
Oct 31 03:39:21.226554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.223715    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" event={"ID":"91d023f6-cc69-4216-9c6b-8dad180323f7","Type":"ContainerDied","Data":"79c5c90fef87af9ffda057786e56284d046fa7af60d00cae6f0910d2ade3ebb5"}
Oct 31 03:39:21.226554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.226145    5831 generic.go:356] "Generic (PLEG): container finished" podID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerID="c7e19798a61c179c4ac40c77dc797645e923109698d14c0427700753818b6c70" exitCode=0
Oct 31 03:39:21.226554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.226376    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" event={"ID":"31e80dea-abb0-403f-b5ab-1ccfb17d811e","Type":"ContainerDied","Data":"c7e19798a61c179c4ac40c77dc797645e923109698d14c0427700753818b6c70"}
Oct 31 03:39:21.239121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.239076    5831 generic.go:356] "Generic (PLEG): container finished" podID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerID="b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb" exitCode=0
Oct 31 03:39:21.239320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.239207    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" event={"ID":"37340dc1-dbe9-4b25-892b-1d44638c5b3f","Type":"ContainerDied","Data":"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"}
Oct 31 03:39:21.239320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.239252    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7" event={"ID":"37340dc1-dbe9-4b25-892b-1d44638c5b3f","Type":"ContainerDied","Data":"57ddae0dd3b38c35c074262b5022ba3a54175112fcf8c6ef95479200fa91b4a6"}
Oct 31 03:39:21.239320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.239282    5831 scope.go:122] "RemoveContainer" containerID="b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"
Oct 31 03:39:21.239504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.239440    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"
Oct 31 03:39:21.267448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.267422    5831 scope.go:122] "RemoveContainer" containerID="b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"
Oct 31 03:39:21.267973 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:21.267948    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb\": container with ID starting with b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb not found: ID does not exist" containerID="b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"
Oct 31 03:39:21.268063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.267977    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb"} err="failed to get container status \"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb\": rpc error: code = NotFound desc = could not find container \"b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb\": container with ID starting with b56712bb073696c4879d7db247d1d9d3d7f753204d58c3b793d0c4613a934bfb not found: ID does not exist"
Oct 31 03:39:21.268646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.268628    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:39:21.277413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277391    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277519    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277585    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277713    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277862    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.277949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277902    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.278130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.277967    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.278130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.278009    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\") pod \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\" (UID: \"37340dc1-dbe9-4b25-892b-1d44638c5b3f\") "
Oct 31 03:39:21.278130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.278094    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:21.287510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.278406    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.287510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.278415    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.287510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.284086    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:21.287510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.287287    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.287510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.287439    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.291075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.290630    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.294549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.291691    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:21.340223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.340164    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "kube-api-access-n6x6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:21.345878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.341688    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy" pod "37340dc1-dbe9-4b25-892b-1d44638c5b3f" (UID: "37340dc1-dbe9-4b25-892b-1d44638c5b3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.368856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.366954    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:21.368856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.367012    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379593    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379690    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379777    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379816    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379867    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379938    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.379968    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.380000    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.380413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.380038    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\") pod \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\" (UID: \"31e80dea-abb0-403f-b5ab-1ccfb17d811e\") "
Oct 31 03:39:21.382275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.381329    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.382275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.381515    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.382275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.381981    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:21.383631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.383273    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384417    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384446    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384460    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384475    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384489    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384504    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.384656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.384518    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.385224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385172    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.385453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385353    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37340dc1-dbe9-4b25-892b-1d44638c5b3f-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.385453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385374    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37340dc1-dbe9-4b25-892b-1d44638c5b3f-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.385453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385388    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.385453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385413    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.385453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.385427    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6x6r\" (UniqueName: \"kubernetes.io/projected/37340dc1-dbe9-4b25-892b-1d44638c5b3f-kube-api-access-n6x6r\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.414007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.410724    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:21.457279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.457210    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "kube-api-access-9zmwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:21.478284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.478225    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:21.478861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.478824    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo" pod "31e80dea-abb0-403f-b5ab-1ccfb17d811e" (UID: "31e80dea-abb0-403f-b5ab-1ccfb17d811e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:21.539997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.488947    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.539997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.488983    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.539997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.488999    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmwc\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-kube-api-access-9zmwc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.539997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.489014    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.539997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.489029    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e80dea-abb0-403f-b5ab-1ccfb17d811e-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:21.767918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.767885    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:39:21.774978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.774943    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h2zq7"]
Oct 31 03:39:21.857595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.857064    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:39:21.965280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.965216    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:39:21.999773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.999349    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 03:39:21.999773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:21.999406    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 03:39:22.042756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.042576    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\" (UniqueName: \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\") pod \"91d023f6-cc69-4216-9c6b-8dad180323f7\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") "
Oct 31 03:39:22.042756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.042693    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\") pod \"91d023f6-cc69-4216-9c6b-8dad180323f7\" (UID: \"91d023f6-cc69-4216-9c6b-8dad180323f7\") "
Oct 31 03:39:22.055812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.045804    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume" pod "91d023f6-cc69-4216-9c6b-8dad180323f7" (UID: "91d023f6-cc69-4216-9c6b-8dad180323f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:22.055812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.052767    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d023f6-cc69-4216-9c6b-8dad180323f7-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.059879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.059465    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk" pod "91d023f6-cc69-4216-9c6b-8dad180323f7" (UID: "91d023f6-cc69-4216-9c6b-8dad180323f7"). InnerVolumeSpecName "kube-api-access-86cdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:22.146778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.146668    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" path="/var/lib/kubelet/pods/37340dc1-dbe9-4b25-892b-1d44638c5b3f/volumes"
Oct 31 03:39:22.153616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.153584    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\") pod \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") "
Oct 31 03:39:22.153775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.153690    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\" (UniqueName: \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\") pod \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\" (UID: \"3b556e4a-215c-4bd6-9e2e-fc954396da0d\") "
Oct 31 03:39:22.154157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.154134    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86cdk\" (UniqueName: \"kubernetes.io/projected/91d023f6-cc69-4216-9c6b-8dad180323f7-kube-api-access-86cdk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.154553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.154507    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume" pod "3b556e4a-215c-4bd6-9e2e-fc954396da0d" (UID: "3b556e4a-215c-4bd6-9e2e-fc954396da0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:22.157702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.157671    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz" pod "3b556e4a-215c-4bd6-9e2e-fc954396da0d" (UID: "3b556e4a-215c-4bd6-9e2e-fc954396da0d"). InnerVolumeSpecName "kube-api-access-fwplz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:22.248506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.248476    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"
Oct 31 03:39:22.249089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.248471    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk" event={"ID":"91d023f6-cc69-4216-9c6b-8dad180323f7","Type":"ContainerDied","Data":"ff8574ca1a8357ef3c3ce202671c1a861be2dfcfa6816e5e37009a4bf1cb76eb"}
Oct 31 03:39:22.249089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.248993    5831 scope.go:122] "RemoveContainer" containerID="79c5c90fef87af9ffda057786e56284d046fa7af60d00cae6f0910d2ade3ebb5"
Oct 31 03:39:22.250626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.250562    5831 generic.go:356] "Generic (PLEG): container finished" podID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" containerID="f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce" exitCode=0
Oct 31 03:39:22.250852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.250830    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"
Oct 31 03:39:22.250936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.250839    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" event={"ID":"3b556e4a-215c-4bd6-9e2e-fc954396da0d","Type":"ContainerDied","Data":"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"}
Oct 31 03:39:22.250981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.250954    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl" event={"ID":"3b556e4a-215c-4bd6-9e2e-fc954396da0d","Type":"ContainerDied","Data":"13c1d9cdb3be04f389cf6907b3cf6ad1d10f7c3b72bf264e067c8ff251451a07"}
Oct 31 03:39:22.253084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.253065    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"
Oct 31 03:39:22.253189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.253093    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88" event={"ID":"31e80dea-abb0-403f-b5ab-1ccfb17d811e","Type":"ContainerDied","Data":"6a7c0297b4a8e02d9ec9da8ef4949c4187b439c0eed9dc1b14e2ace2694051be"}
Oct 31 03:39:22.255549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.255502    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b556e4a-215c-4bd6-9e2e-fc954396da0d-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.255635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.255567    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwplz\" (UniqueName: \"kubernetes.io/projected/3b556e4a-215c-4bd6-9e2e-fc954396da0d-kube-api-access-fwplz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.262662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.262645    5831 scope.go:122] "RemoveContainer" containerID="f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"
Oct 31 03:39:22.272613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.272600    5831 scope.go:122] "RemoveContainer" containerID="f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"
Oct 31 03:39:22.272933 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:22.272913    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce\": container with ID starting with f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce not found: ID does not exist" containerID="f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"
Oct 31 03:39:22.272996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.272941    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce"} err="failed to get container status \"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce\": rpc error: code = NotFound desc = could not find container \"f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce\": container with ID starting with f9b44a2fdacf0144b0c17d4b3615e30119cdc70016b6abf1f28b7dcb970610ce not found: ID does not exist"
Oct 31 03:39:22.272996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.272958    5831 scope.go:122] "RemoveContainer" containerID="c7e19798a61c179c4ac40c77dc797645e923109698d14c0427700753818b6c70"
Oct 31 03:39:22.363135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.363103    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:22.363294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.363159    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:22.431678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.431595    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:39:22.436080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.436057    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-7d7b895cc5-mfnpk"]
Oct 31 03:39:22.571734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.571693    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:39:22.600558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.590940    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rpg88"]
Oct 31 03:39:22.686665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.686575    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:39:22.715569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.709660    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-855c78f44c-fs9dl"]
Oct 31 03:39:23.000618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.997972    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 03:39:23.000618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:22.998037    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 03:39:23.362781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:23.362672    5831 patch_prober.go:29] interesting pod/e2e-api-a-329d79-openshift-default-575867c7-zmd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 31 03:39:23.362781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:23.362731    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 31 03:39:23.999809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:23.999773    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:39:24.000302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.000282    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:39:24.001009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.000990    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:39:24.137435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.137386    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" path="/var/lib/kubelet/pods/31e80dea-abb0-403f-b5ab-1ccfb17d811e/volumes"
Oct 31 03:39:24.137920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.137902    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" path="/var/lib/kubelet/pods/3b556e4a-215c-4bd6-9e2e-fc954396da0d/volumes"
Oct 31 03:39:24.138295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.138281    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d023f6-cc69-4216-9c6b-8dad180323f7" path="/var/lib/kubelet/pods/91d023f6-cc69-4216-9c6b-8dad180323f7/volumes"
Oct 31 03:39:24.368172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.368069    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:39:24.369196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:24.369174    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:39:34.500204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:34.500174    5831 scope.go:122] "RemoveContainer" containerID="53c11ce0ac2f52653b4767c31feb0cd997569d0757d8f224a7430db9f6ae8a8a"
Oct 31 03:39:39.601776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:39.601726    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:39:39.602221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:39.601932    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" podUID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" containerName="maas-api" containerID="cri-o://9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e" gracePeriod=30
Oct 31 03:39:39.948627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:39.948600    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:39:40.097040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.097006    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\") pod \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") "
Oct 31 03:39:40.100329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.098445    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") pod \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") "
Oct 31 03:39:40.100329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.098523    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\") pod \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") "
Oct 31 03:39:40.100329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.098682    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\" (UniqueName: \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\") pod \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\" (UID: \"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16\") "
Oct 31 03:39:40.100329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.100024    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca" pod "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" (UID: "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:40.103178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.103147    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:39:40.115098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.110389    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5" pod "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" (UID: "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16"). InnerVolumeSpecName "kube-api-access-7msl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:40.115098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.110586    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls" pod "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" (UID: "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:39:40.115098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.112760    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs" pod "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" (UID: "7746c1ee-b1e3-4bd3-9882-ab0f9168ae16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:39:40.115955 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.115925    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" podUID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" containerName="payload-processing" containerID="cri-o://c83befe83207d4bcc35bf34c4aa23a5dd88ccee338d4fef0c97727fc2de69552" gracePeriod=30
Oct 31 03:39:40.142886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.142859    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:39:40.146693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.146605    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" podUID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" containerName="payload-pre-processing" containerID="cri-o://fb9107f156974c5643a7e9029f4d2253cdc8e886453315f11a91f8862f712313" gracePeriod=30
Oct 31 03:39:40.216586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.206877    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msl5\" (UniqueName: \"kubernetes.io/projected/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-kube-api-access-7msl5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.216586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.206914    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.216586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.206929    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.216586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.206943    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.431045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.430999    5831 generic.go:356] "Generic (PLEG): container finished" podID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" containerID="c83befe83207d4bcc35bf34c4aa23a5dd88ccee338d4fef0c97727fc2de69552" exitCode=0
Oct 31 03:39:40.431515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.431480    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" event={"ID":"29b0d556-17be-49d2-8bef-ac6e69cc36c4","Type":"ContainerDied","Data":"c83befe83207d4bcc35bf34c4aa23a5dd88ccee338d4fef0c97727fc2de69552"}
Oct 31 03:39:40.434994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.434962    5831 generic.go:356] "Generic (PLEG): container finished" podID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" containerID="fb9107f156974c5643a7e9029f4d2253cdc8e886453315f11a91f8862f712313" exitCode=0
Oct 31 03:39:40.435100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.435068    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" event={"ID":"d02ad148-a71b-4078-8eb3-d9bb4f2319ed","Type":"ContainerDied","Data":"fb9107f156974c5643a7e9029f4d2253cdc8e886453315f11a91f8862f712313"}
Oct 31 03:39:40.436637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.436613    5831 generic.go:356] "Generic (PLEG): container finished" podID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" containerID="9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e" exitCode=0
Oct 31 03:39:40.436755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.436710    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" event={"ID":"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16","Type":"ContainerDied","Data":"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"}
Oct 31 03:39:40.436827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.436751    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29" event={"ID":"7746c1ee-b1e3-4bd3-9882-ab0f9168ae16","Type":"ContainerDied","Data":"39ad1d2efe1bc729391514baa25aa6e99a40adf7012c4345adda0fd4ef7ef99b"}
Oct 31 03:39:40.436827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.436782    5831 scope.go:122] "RemoveContainer" containerID="9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"
Oct 31 03:39:40.436977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.436962    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"
Oct 31 03:39:40.449792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.449770    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:39:40.457669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.457642    5831 scope.go:122] "RemoveContainer" containerID="9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"
Oct 31 03:39:40.458040 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:40.458019    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e\": container with ID starting with 9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e not found: ID does not exist" containerID="9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"
Oct 31 03:39:40.458121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.458049    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e"} err="failed to get container status \"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e\": rpc error: code = NotFound desc = could not find container \"9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e\": container with ID starting with 9edefe02f5ed3a59c19ccbc57ab89b7ad0a155965831a1938e8e69e3001de20e not found: ID does not exist"
Oct 31 03:39:40.478292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.478266    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:39:40.631958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.630106    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\" (UniqueName: \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\") pod \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") "
Oct 31 03:39:40.631958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.630163    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\") pod \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\" (UID: \"29b0d556-17be-49d2-8bef-ac6e69cc36c4\") "
Oct 31 03:39:40.631958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.630252    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\" (UniqueName: \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\") pod \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") "
Oct 31 03:39:40.631958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.630393    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\") pod \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\" (UID: \"d02ad148-a71b-4078-8eb3-d9bb4f2319ed\") "
Oct 31 03:39:40.631958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.631256    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume" pod "d02ad148-a71b-4078-8eb3-d9bb4f2319ed" (UID: "d02ad148-a71b-4078-8eb3-d9bb4f2319ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:40.651806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.651772    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume" pod "29b0d556-17be-49d2-8bef-ac6e69cc36c4" (UID: "29b0d556-17be-49d2-8bef-ac6e69cc36c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:40.715808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.710210    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb" pod "d02ad148-a71b-4078-8eb3-d9bb4f2319ed" (UID: "d02ad148-a71b-4078-8eb3-d9bb4f2319ed"). InnerVolumeSpecName "kube-api-access-h5swb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:40.715808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.715749    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn" pod "29b0d556-17be-49d2-8bef-ac6e69cc36c4" (UID: "29b0d556-17be-49d2-8bef-ac6e69cc36c4"). InnerVolumeSpecName "kube-api-access-644kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:40.768219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.762018    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5swb\" (UniqueName: \"kubernetes.io/projected/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-kube-api-access-h5swb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.768219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.762062    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02ad148-a71b-4078-8eb3-d9bb4f2319ed-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.768219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.762076    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644kn\" (UniqueName: \"kubernetes.io/projected/29b0d556-17be-49d2-8bef-ac6e69cc36c4-kube-api-access-644kn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.768219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.762090    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b0d556-17be-49d2-8bef-ac6e69cc36c4-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:40.784966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.784874    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:39:40.788307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.788276    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d3f8a-df788bbdf-mxz29"]
Oct 31 03:39:40.871441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.871409    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:39:40.871681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:40.871653    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" podUID="2974f595-4c3d-434a-95e9-95ccf152f9b4" containerName="maas-api" containerID="cri-o://15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618" gracePeriod=30
Oct 31 03:39:41.210805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.210774    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:39:41.385724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.384455    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\") pod \"2974f595-4c3d-434a-95e9-95ccf152f9b4\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") "
Oct 31 03:39:41.385724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.384566    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\") pod \"2974f595-4c3d-434a-95e9-95ccf152f9b4\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") "
Oct 31 03:39:41.385724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.384607    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\" (UniqueName: \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\") pod \"2974f595-4c3d-434a-95e9-95ccf152f9b4\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") "
Oct 31 03:39:41.385724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.384809    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") pod \"2974f595-4c3d-434a-95e9-95ccf152f9b4\" (UID: \"2974f595-4c3d-434a-95e9-95ccf152f9b4\") "
Oct 31 03:39:41.390045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.388607    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca" pod "2974f595-4c3d-434a-95e9-95ccf152f9b4" (UID: "2974f595-4c3d-434a-95e9-95ccf152f9b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:41.391794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.391753    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n" pod "2974f595-4c3d-434a-95e9-95ccf152f9b4" (UID: "2974f595-4c3d-434a-95e9-95ccf152f9b4"). InnerVolumeSpecName "kube-api-access-nnl7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:41.393154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.393111    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls" pod "2974f595-4c3d-434a-95e9-95ccf152f9b4" (UID: "2974f595-4c3d-434a-95e9-95ccf152f9b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:39:41.393475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.393446    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs" pod "2974f595-4c3d-434a-95e9-95ccf152f9b4" (UID: "2974f595-4c3d-434a-95e9-95ccf152f9b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:39:41.398549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.398507    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:39:41.398819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.398771    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" podUID="8f269c40-4e5e-447c-a794-f24aceb3762e" containerName="payload-processing" containerID="cri-o://575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729" gracePeriod=30
Oct 31 03:39:41.420574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.420415    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:39:41.420842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.420811    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" podUID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" containerName="payload-pre-processing" containerID="cri-o://224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b" gracePeriod=30
Oct 31 03:39:41.446746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.446713    5831 generic.go:356] "Generic (PLEG): container finished" podID="2974f595-4c3d-434a-95e9-95ccf152f9b4" containerID="15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618" exitCode=0
Oct 31 03:39:41.446886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.446804    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"
Oct 31 03:39:41.446886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.446804    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" event={"ID":"2974f595-4c3d-434a-95e9-95ccf152f9b4","Type":"ContainerDied","Data":"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"}
Oct 31 03:39:41.446886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.446862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr" event={"ID":"2974f595-4c3d-434a-95e9-95ccf152f9b4","Type":"ContainerDied","Data":"43bc6f777f663b04ac0903ccaf0136caa0a6a6cf9248a1c6d4fc658329d9f379"}
Oct 31 03:39:41.447063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.446889    5831 scope.go:122] "RemoveContainer" containerID="15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"
Oct 31 03:39:41.448919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.448894    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"
Oct 31 03:39:41.449023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.448906    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4" event={"ID":"29b0d556-17be-49d2-8bef-ac6e69cc36c4","Type":"ContainerDied","Data":"d306c3576d782163f96b4b879c5aa666e481fa258da8b0ee1d9faf34547b4f40"}
Oct 31 03:39:41.450690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.450666    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9" event={"ID":"d02ad148-a71b-4078-8eb3-d9bb4f2319ed","Type":"ContainerDied","Data":"c04c8c1ef63e781cf8526302f98c106d1f27713fcddd15d6fe27b62ff667fd12"}
Oct 31 03:39:41.450781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.450746    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"
Oct 31 03:39:41.486550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.486505    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974f595-4c3d-434a-95e9-95ccf152f9b4-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.486749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.486731    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.486848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.486824    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnl7n\" (UniqueName: \"kubernetes.io/projected/2974f595-4c3d-434a-95e9-95ccf152f9b4-kube-api-access-nnl7n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.486919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.486858    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974f595-4c3d-434a-95e9-95ccf152f9b4-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.543197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.543165    5831 scope.go:122] "RemoveContainer" containerID="15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"
Oct 31 03:39:41.546421 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:41.543613    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618\": container with ID starting with 15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618 not found: ID does not exist" containerID="15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"
Oct 31 03:39:41.546421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.543657    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618"} err="failed to get container status \"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618\": rpc error: code = NotFound desc = could not find container \"15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618\": container with ID starting with 15b818618cfa5bac6e8d739a00a8ad35ae2396aa1e0302e8d0e312a8c557d618 not found: ID does not exist"
Oct 31 03:39:41.546421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.543683    5831 scope.go:122] "RemoveContainer" containerID="c83befe83207d4bcc35bf34c4aa23a5dd88ccee338d4fef0c97727fc2de69552"
Oct 31 03:39:41.561687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.561666    5831 scope.go:122] "RemoveContainer" containerID="fb9107f156974c5643a7e9029f4d2253cdc8e886453315f11a91f8862f712313"
Oct 31 03:39:41.613969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.613937    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:39:41.619234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.619187    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d3f8a-5fcd98d775-rb9n9"]
Oct 31 03:39:41.717238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.717201    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:39:41.724545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.724499    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d3f8a-546658988d-4msn4"]
Oct 31 03:39:41.746137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.746105    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:39:41.787590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.787555    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:39:41.801346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.801310    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\" (UniqueName: \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\") pod \"8f269c40-4e5e-447c-a794-f24aceb3762e\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") "
Oct 31 03:39:41.801550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.801424    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\") pod \"8f269c40-4e5e-447c-a794-f24aceb3762e\" (UID: \"8f269c40-4e5e-447c-a794-f24aceb3762e\") "
Oct 31 03:39:41.801820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.801786    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume" pod "8f269c40-4e5e-447c-a794-f24aceb3762e" (UID: "8f269c40-4e5e-447c-a794-f24aceb3762e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:41.802171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.802148    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f269c40-4e5e-447c-a794-f24aceb3762e-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.803809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.803781    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8" pod "8f269c40-4e5e-447c-a794-f24aceb3762e" (UID: "8f269c40-4e5e-447c-a794-f24aceb3762e"). InnerVolumeSpecName "kube-api-access-j9sr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:41.823336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.823306    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:39:41.829045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.829019    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1aad0-7944fc79f8-qmrzr"]
Oct 31 03:39:41.909593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.906357    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\" (UniqueName: \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\") pod \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") "
Oct 31 03:39:41.909593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.906652    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\") pod \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\" (UID: \"1760af4c-e52e-42fa-8ba0-8018e70c6a5e\") "
Oct 31 03:39:41.909593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.907194    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9sr8\" (UniqueName: \"kubernetes.io/projected/8f269c40-4e5e-447c-a794-f24aceb3762e-kube-api-access-j9sr8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:41.919652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.913777    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume" pod "1760af4c-e52e-42fa-8ba0-8018e70c6a5e" (UID: "1760af4c-e52e-42fa-8ba0-8018e70c6a5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:41.928554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:41.927110    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8" pod "1760af4c-e52e-42fa-8ba0-8018e70c6a5e" (UID: "1760af4c-e52e-42fa-8ba0-8018e70c6a5e"). InnerVolumeSpecName "kube-api-access-59gp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:42.008874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.008844    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gp8\" (UniqueName: \"kubernetes.io/projected/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-kube-api-access-59gp8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:42.009115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.009093    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1760af4c-e52e-42fa-8ba0-8018e70c6a5e-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:42.137085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.137050    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2974f595-4c3d-434a-95e9-95ccf152f9b4" path="/var/lib/kubelet/pods/2974f595-4c3d-434a-95e9-95ccf152f9b4/volumes"
Oct 31 03:39:42.137491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.137477    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" path="/var/lib/kubelet/pods/29b0d556-17be-49d2-8bef-ac6e69cc36c4/volumes"
Oct 31 03:39:42.137965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.137949    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" path="/var/lib/kubelet/pods/7746c1ee-b1e3-4bd3-9882-ab0f9168ae16/volumes"
Oct 31 03:39:42.138695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.138674    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" path="/var/lib/kubelet/pods/d02ad148-a71b-4078-8eb3-d9bb4f2319ed/volumes"
Oct 31 03:39:42.459271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.459231    5831 generic.go:356] "Generic (PLEG): container finished" podID="8f269c40-4e5e-447c-a794-f24aceb3762e" containerID="575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729" exitCode=0
Oct 31 03:39:42.459484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.459275    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"
Oct 31 03:39:42.459484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.459320    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" event={"ID":"8f269c40-4e5e-447c-a794-f24aceb3762e","Type":"ContainerDied","Data":"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"}
Oct 31 03:39:42.459484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.459375    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc" event={"ID":"8f269c40-4e5e-447c-a794-f24aceb3762e","Type":"ContainerDied","Data":"5ff6315d2d174e97ba59d64ac7097dc6a4eddbce734bbfef510721ed803df3e0"}
Oct 31 03:39:42.459484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.459403    5831 scope.go:122] "RemoveContainer" containerID="575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"
Oct 31 03:39:42.464152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.464126    5831 generic.go:356] "Generic (PLEG): container finished" podID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" containerID="224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b" exitCode=0
Oct 31 03:39:42.464263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.464170    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"
Oct 31 03:39:42.464263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.464204    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" event={"ID":"1760af4c-e52e-42fa-8ba0-8018e70c6a5e","Type":"ContainerDied","Data":"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"}
Oct 31 03:39:42.464263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.464247    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz" event={"ID":"1760af4c-e52e-42fa-8ba0-8018e70c6a5e","Type":"ContainerDied","Data":"3835360c008408b7b1a51e8f0b4c4c4903e46a1733e6bb517bbfb56a758938ac"}
Oct 31 03:39:42.470492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.470472    5831 scope.go:122] "RemoveContainer" containerID="575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"
Oct 31 03:39:42.470818 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:42.470799    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729\": container with ID starting with 575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729 not found: ID does not exist" containerID="575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"
Oct 31 03:39:42.470861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.470826    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729"} err="failed to get container status \"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729\": rpc error: code = NotFound desc = could not find container \"575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729\": container with ID starting with 575b2b60b2aae55fb5d1599d5908745192536c78867c832e53d9ee9bd333a729 not found: ID does not exist"
Oct 31 03:39:42.470861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.470842    5831 scope.go:122] "RemoveContainer" containerID="224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"
Oct 31 03:39:42.480465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.480445    5831 scope.go:122] "RemoveContainer" containerID="224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"
Oct 31 03:39:42.480798 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:42.480768    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b\": container with ID starting with 224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b not found: ID does not exist" containerID="224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"
Oct 31 03:39:42.480935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.480805    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b"} err="failed to get container status \"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b\": rpc error: code = NotFound desc = could not find container \"224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b\": container with ID starting with 224497df9cc17243abfc8e3e1ecc870c5c4bd37c01ca153098575ccc2faa100b not found: ID does not exist"
Oct 31 03:39:42.613700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.613663    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:39:42.617662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.617637    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1aad0-557979479d-49tzc"]
Oct 31 03:39:42.723184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.723085    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:39:42.726403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:42.726373    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1aad0-6f9d5fd4c9-x4whz"]
Oct 31 03:39:42.978556 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:42.978453    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:39:44.136880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:44.136804    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" path="/var/lib/kubelet/pods/1760af4c-e52e-42fa-8ba0-8018e70c6a5e/volumes"
Oct 31 03:39:44.137254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:44.137185    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f269c40-4e5e-447c-a794-f24aceb3762e" path="/var/lib/kubelet/pods/8f269c40-4e5e-447c-a794-f24aceb3762e/volumes"
Oct 31 03:39:47.766332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:47.766264    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:39:47.767203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:47.767168    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy" containerID="cri-o://5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993" gracePeriod=30
Oct 31 03:39:48.499594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.499555    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:39:48.500317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500289    5831 cpu_manager.go:414] "containerMap: removing container" podUID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" containerName="payload-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500315    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" containerName="payload-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500339    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" containerName="maas-api"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500346    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" containerName="maas-api"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500362    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2974f595-4c3d-434a-95e9-95ccf152f9b4" containerName="maas-api"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500369    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2974f595-4c3d-434a-95e9-95ccf152f9b4" containerName="maas-api"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500387    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8f269c40-4e5e-447c-a794-f24aceb3762e" containerName="payload-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500393    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f269c40-4e5e-447c-a794-f24aceb3762e" containerName="payload-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500412    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500418    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500430    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500437    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500450    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500457    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" containerName="payload-pre-processing"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500482    5831 cpu_manager.go:414] "containerMap: removing container" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy"
Oct 31 03:39:48.500482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500486    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500499    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dc081e01-3334-46d1-928b-6ff0a2c494fd" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500504    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc081e01-3334-46d1-928b-6ff0a2c494fd" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500514    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500520    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500551    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f0514f03-0b20-422a-8630-ee4cf0c696dd" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500558    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0514f03-0b20-422a-8630-ee4cf0c696dd" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500568    5831 cpu_manager.go:414] "containerMap: removing container" podUID="91d023f6-cc69-4216-9c6b-8dad180323f7" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500574    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d023f6-cc69-4216-9c6b-8dad180323f7" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500591    5831 cpu_manager.go:414] "containerMap: removing container" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500600    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500612    5831 cpu_manager.go:414] "containerMap: removing container" podUID="23c47c73-ddc8-4218-92d9-26b83d21507b" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500618    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c47c73-ddc8-4218-92d9-26b83d21507b" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500761    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02ad148-a71b-4078-8eb3-d9bb4f2319ed" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500778    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0514f03-0b20-422a-8630-ee4cf0c696dd" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500788    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d023f6-cc69-4216-9c6b-8dad180323f7" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500797    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e80dea-abb0-403f-b5ab-1ccfb17d811e" containerName="istio-proxy"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500806    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c47c73-ddc8-4218-92d9-26b83d21507b" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500815    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b0d556-17be-49d2-8bef-ac6e69cc36c4" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500826    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7746c1ee-b1e3-4bd3-9882-ab0f9168ae16" containerName="maas-api"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500837    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2974f595-4c3d-434a-95e9-95ccf152f9b4" containerName="maas-api"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500845    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f269c40-4e5e-447c-a794-f24aceb3762e" containerName="payload-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500854    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1760af4c-e52e-42fa-8ba0-8018e70c6a5e" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500861    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4b6b9d-87e7-4358-8a37-6233a59fbb77" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500870    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b556e4a-215c-4bd6-9e2e-fc954396da0d" containerName="payload-pre-processing"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500883    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="37340dc1-dbe9-4b25-892b-1d44638c5b3f" containerName="istio-proxy"
Oct 31 03:39:48.501169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.500893    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc081e01-3334-46d1-928b-6ff0a2c494fd" containerName="payload-processing"
Oct 31 03:39:48.505903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.505870    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.559175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.559146    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:39:48.601187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.601152    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-18d5f6\""
Oct 31 03:39:48.601641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.601622    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-18d5f6-dockercfg-xxpqr\""
Oct 31 03:39:48.627700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.627667    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gkh\" (UniqueName: \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.627869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.627818    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.728964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.728916    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.729147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.729129    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gkh\" (UniqueName: \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.729734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.729713    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.782795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.782712    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gkh\" (UniqueName: \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\") pod \"payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:48.821665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:48.821627    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:49.196936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.196900    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:39:49.203443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.203415    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:39:49.203592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.203518    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.233976 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:39:49.233943    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac0273d_8a7b_4d37_984e_6aedc278a6d3.slice/crio-fd9b4e68ea7768d06434cee5be57267c56ce9f46169b6a1dc629eea894b7f6ac WatchSource:0}: Error finding container fd9b4e68ea7768d06434cee5be57267c56ce9f46169b6a1dc629eea894b7f6ac: Status 404 returned error can't find the container with id fd9b4e68ea7768d06434cee5be57267c56ce9f46169b6a1dc629eea894b7f6ac
Oct 31 03:39:49.234552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.234511    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lnj\" (UniqueName: \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.234747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.234720    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.254252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.254213    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:39:49.335745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.335712    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lnj\" (UniqueName: \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.335909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.335753    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.336469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.336440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.388446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.388421    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lnj\" (UniqueName: \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\") pod \"payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.519348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.519251    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:49.538451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.538420    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" event={"ID":"7ac0273d-8a7b-4d37-984e-6aedc278a6d3","Type":"ContainerStarted","Data":"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"}
Oct 31 03:39:49.538627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.538479    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" event={"ID":"7ac0273d-8a7b-4d37-984e-6aedc278a6d3","Type":"ContainerStarted","Data":"fd9b4e68ea7768d06434cee5be57267c56ce9f46169b6a1dc629eea894b7f6ac"}
Oct 31 03:39:49.538627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.538509    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:39:49.649421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.649359    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" podStartSLOduration=1.649344663 podStartE2EDuration="1.649344663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:49.647111713 +0000 UTC m=+1939.661673421" watchObservedRunningTime="2026-10-31 03:39:49.649344663 +0000 UTC m=+1939.663906374"
Oct 31 03:39:49.936753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:49.936725    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:39:49.938634 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:39:49.938602    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd3ec40_85c4_45bf_8326_00e7601bb237.slice/crio-a0085bf2ac7dbe3ce634133b63e7a492d3c90da933857905b987fd96a1b877e5 WatchSource:0}: Error finding container a0085bf2ac7dbe3ce634133b63e7a492d3c90da933857905b987fd96a1b877e5: Status 404 returned error can't find the container with id a0085bf2ac7dbe3ce634133b63e7a492d3c90da933857905b987fd96a1b877e5
Oct 31 03:39:50.154143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.154107    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:39:50.158871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.158843    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.208203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.208090    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c35264-serving-cert\""
Oct 31 03:39:50.226050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.226021    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:39:50.246272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.246245    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.246450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.246318    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.246519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.246446    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.246720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.246692    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7wn\" (UniqueName: \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.347992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.347960    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7wn\" (UniqueName: \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.348178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.348084    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.348178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.348140    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.348328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.348194    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.348406 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:50.348377    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-serving-cert: secret "maas-api-e2e-ipp-b-c35264-serving-cert" not found
Oct 31 03:39:50.348471 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:50.348453    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls podName:ba3d7995-7216-478e-8244-e0bcab6f5145 nodeName:}" failed. No retries permitted until 2026-10-31 03:39:50.848431978 +0000 UTC m=+1940.862993678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls") pod "maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" (UID: "ba3d7995-7216-478e-8244-e0bcab6f5145") : secret "maas-api-e2e-ipp-b-c35264-serving-cert" not found
Oct 31 03:39:50.349044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.349008    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.350929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.350907    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.404358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.404315    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7wn\" (UniqueName: \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.478683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.478583    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:39:50.478862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.478834    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" containerID="cri-o://7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25" gracePeriod=30
Oct 31 03:39:50.482179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.482151    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:39:50.482465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.482366    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" containerID="cri-o://13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249" gracePeriod=30
Oct 31 03:39:50.548787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.548748    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" event={"ID":"5cd3ec40-85c4-45bf-8326-00e7601bb237","Type":"ContainerStarted","Data":"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"}
Oct 31 03:39:50.548970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.548809    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" event={"ID":"5cd3ec40-85c4-45bf-8326-00e7601bb237","Type":"ContainerStarted","Data":"a0085bf2ac7dbe3ce634133b63e7a492d3c90da933857905b987fd96a1b877e5"}
Oct 31 03:39:50.548970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.548866    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:39:50.863111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.862588    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:50.863597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.863518    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" podStartSLOduration=1.8635011000000001 podStartE2EDuration="1.8635011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:50.749255192 +0000 UTC m=+1940.763816900" watchObservedRunningTime="2026-10-31 03:39:50.8635011 +0000 UTC m=+1940.878062809"
Oct 31 03:39:50.887553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:50.885512    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:51.076038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:51.076001    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:51.461344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:51.461310    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:39:51.463472 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:39:51.463446    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3d7995_7216_478e_8244_e0bcab6f5145.slice/crio-608bc5679f2fc46ac97848a9d9af6aa94393e05bd2f6b76dbd8bf948d06acc03 WatchSource:0}: Error finding container 608bc5679f2fc46ac97848a9d9af6aa94393e05bd2f6b76dbd8bf948d06acc03: Status 404 returned error can't find the container with id 608bc5679f2fc46ac97848a9d9af6aa94393e05bd2f6b76dbd8bf948d06acc03
Oct 31 03:39:51.464947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:51.464924    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:39:51.557392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:51.557352    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" event={"ID":"ba3d7995-7216-478e-8244-e0bcab6f5145","Type":"ContainerStarted","Data":"608bc5679f2fc46ac97848a9d9af6aa94393e05bd2f6b76dbd8bf948d06acc03"}
Oct 31 03:39:52.567945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:52.567904    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" event={"ID":"ba3d7995-7216-478e-8244-e0bcab6f5145","Type":"ContainerStarted","Data":"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"}
Oct 31 03:39:52.568356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:52.568145    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:39:52.687233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:52.687177    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" podStartSLOduration=1.732380043 podStartE2EDuration="2.687157618s" totalImagesPullingTime="954.777575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:52.683402273 +0000 UTC m=+1942.697963984" watchObservedRunningTime="2026-10-31 03:39:52.687157618 +0000 UTC m=+1942.701719326"
Oct 31 03:39:53.099420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.099391    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:39:53.116680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.116645    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:53.116867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.116711    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:53.201357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201317    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.201644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201375    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.201644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201403    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.201644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201462    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.201855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201694    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:53.201855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201724    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.201855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201757    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:53.201855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201787    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.202069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201860    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.202069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201893    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.202069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.201981    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\") pod \"e7181a37-6a28-4eb1-8005-05fe31a01474\" (UID: \"e7181a37-6a28-4eb1-8005-05fe31a01474\") "
Oct 31 03:39:53.202069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.202060    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:53.202277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.202214    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:53.202277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.202261    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:53.203409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.203078    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.203409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.203105    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.203409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.203121    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.203409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.203136    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7181a37-6a28-4eb1-8005-05fe31a01474-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.203409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.203151    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.204739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.204696    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:53.204888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.204759    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:53.207015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.206990    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:53.207130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.207109    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd" pod "e7181a37-6a28-4eb1-8005-05fe31a01474" (UID: "e7181a37-6a28-4eb1-8005-05fe31a01474"). InnerVolumeSpecName "kube-api-access-9wlcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:53.304163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.304124    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.304163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.304152    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.304163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.304163    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wlcd\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-kube-api-access-9wlcd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.304398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.304178    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7181a37-6a28-4eb1-8005-05fe31a01474-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:53.576993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.576910    5831 generic.go:356] "Generic (PLEG): container finished" podID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerID="5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993" exitCode=0
Oct 31 03:39:53.576993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.576973    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"
Oct 31 03:39:53.577520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.576993    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerDied","Data":"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"}
Oct 31 03:39:53.577520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.577052    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb" event={"ID":"e7181a37-6a28-4eb1-8005-05fe31a01474","Type":"ContainerDied","Data":"37302dcf5157cab23c881993026a527b8c32f261c865cc735d1b7710b8946bfc"}
Oct 31 03:39:53.577520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.577077    5831 scope.go:122] "RemoveContainer" containerID="5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"
Oct 31 03:39:53.590202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.590180    5831 scope.go:122] "RemoveContainer" containerID="b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"
Oct 31 03:39:53.602980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.602957    5831 scope.go:122] "RemoveContainer" containerID="5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"
Oct 31 03:39:53.603305 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:53.603286    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993\": container with ID starting with 5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993 not found: ID does not exist" containerID="5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"
Oct 31 03:39:53.603382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.603315    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993"} err="failed to get container status \"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993\": rpc error: code = NotFound desc = could not find container \"5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993\": container with ID starting with 5d5266a4f4fd3a9033be801855693ac7199e468ef45d38112d9f108432e09993 not found: ID does not exist"
Oct 31 03:39:53.603382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.603332    5831 scope.go:122] "RemoveContainer" containerID="b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"
Oct 31 03:39:53.603648 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:53.603626    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168\": container with ID starting with b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168 not found: ID does not exist" containerID="b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"
Oct 31 03:39:53.603721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.603657    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168"} err="failed to get container status \"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168\": rpc error: code = NotFound desc = could not find container \"b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168\": container with ID starting with b27fde135d7ea72eab5a52d23b5026a10a80fb43ba28a85f392a082eddf9a168 not found: ID does not exist"
Oct 31 03:39:53.730567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.730514    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:39:53.776663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:53.776613    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d3f8a-openshift-default-ff5c48f7-5wtgb"]
Oct 31 03:39:54.136351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:54.136322    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" path="/var/lib/kubelet/pods/e7181a37-6a28-4eb1-8005-05fe31a01474/volumes"
Oct 31 03:39:54.317042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:54.317004    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:39:54.611817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:54.611776    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:54.612224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:54.611844    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:55.782761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.782733    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:39:55.856263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.856237    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:39:55.953089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953050    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953123    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953174    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953206    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953241    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953271    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953345    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953442    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953472    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953587    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953639    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953680    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953708    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.953850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953743    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.953850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953781    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.954120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953853    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.954120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953916    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\") pod \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\" (UID: \"97432e9c-c79c-4bab-8cba-d9aa0eb475d3\") "
Oct 31 03:39:55.954120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.953952    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\") pod \"d421a5c4-f284-4f4b-8904-41837c250ead\" (UID: \"d421a5c4-f284-4f4b-8904-41837c250ead\") "
Oct 31 03:39:55.954311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954283    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.954405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954380    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:55.954596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954577    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421a5c4-f284-4f4b-8904-41837c250ead-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.954683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954603    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.954879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954849    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.954979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.954919    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.955108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.955085    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.955974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.955707    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.955974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.955858    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:55.956162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.956115    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.956316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.956276    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.956316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.956293    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.958073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.958047    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.959427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.959399    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.959849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.959829    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "kube-api-access-w59m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.960212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.960186    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "kube-api-access-v6n8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.960315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.960293    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.960446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.960419    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo" pod "97432e9c-c79c-4bab-8cba-d9aa0eb475d3" (UID: "97432e9c-c79c-4bab-8cba-d9aa0eb475d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:55.960557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.960509    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.960696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:55.960675    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo" pod "d421a5c4-f284-4f4b-8904-41837c250ead" (UID: "d421a5c4-f284-4f4b-8904-41837c250ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060745    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060777    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060790    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060802    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060814    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060826    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.060839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060838    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060853    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59m6\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-kube-api-access-w59m6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060866    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421a5c4-f284-4f4b-8904-41837c250ead-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060878    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060889    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060900    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060912    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421a5c4-f284-4f4b-8904-41837c250ead-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060924    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6n8f\" (UniqueName: \"kubernetes.io/projected/d421a5c4-f284-4f4b-8904-41837c250ead-kube-api-access-v6n8f\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060935    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.061336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.060947    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97432e9c-c79c-4bab-8cba-d9aa0eb475d3-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.604921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.604879    5831 generic.go:356] "Generic (PLEG): container finished" podID="d421a5c4-f284-4f4b-8904-41837c250ead" containerID="13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249" exitCode=0
Oct 31 03:39:56.605087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.604944    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"
Oct 31 03:39:56.605087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.604951    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerDied","Data":"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"}
Oct 31 03:39:56.605087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.605009    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf" event={"ID":"d421a5c4-f284-4f4b-8904-41837c250ead","Type":"ContainerDied","Data":"dd1da56f56c110630ccbada259bfd109beb040e8c47baa7bb156a88d54c95bb3"}
Oct 31 03:39:56.605087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.605039    5831 scope.go:122] "RemoveContainer" containerID="13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"
Oct 31 03:39:56.606945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.606920    5831 generic.go:356] "Generic (PLEG): container finished" podID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerID="7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25" exitCode=0
Oct 31 03:39:56.607056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.606948    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" event={"ID":"97432e9c-c79c-4bab-8cba-d9aa0eb475d3","Type":"ContainerDied","Data":"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"}
Oct 31 03:39:56.607056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.607006    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm" event={"ID":"97432e9c-c79c-4bab-8cba-d9aa0eb475d3","Type":"ContainerDied","Data":"5fe8d803b8aba992d78047a5cbefe906b0779ef127d0190d02b8e0468423274f"}
Oct 31 03:39:56.607618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.607600    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"
Oct 31 03:39:56.616177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.616152    5831 scope.go:122] "RemoveContainer" containerID="7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"
Oct 31 03:39:56.625822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.625802    5831 scope.go:122] "RemoveContainer" containerID="13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"
Oct 31 03:39:56.626115 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:56.626086    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249\": container with ID starting with 13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249 not found: ID does not exist" containerID="13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"
Oct 31 03:39:56.626170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.626115    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249"} err="failed to get container status \"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249\": rpc error: code = NotFound desc = could not find container \"13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249\": container with ID starting with 13807ed60bf6212bb241133ed82c085b5782a800c328b5e6de15618ee3c30249 not found: ID does not exist"
Oct 31 03:39:56.626170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.626132    5831 scope.go:122] "RemoveContainer" containerID="7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"
Oct 31 03:39:56.626429 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:56.626407    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e\": container with ID starting with 7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e not found: ID does not exist" containerID="7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"
Oct 31 03:39:56.626553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.626432    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e"} err="failed to get container status \"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e\": rpc error: code = NotFound desc = could not find container \"7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e\": container with ID starting with 7c3a41bee8cc6164529764fd24a03b836161b7b883438613261eceaf3bd2a34e not found: ID does not exist"
Oct 31 03:39:56.626553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.626450    5831 scope.go:122] "RemoveContainer" containerID="7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"
Oct 31 03:39:56.636300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.636284    5831 scope.go:122] "RemoveContainer" containerID="7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"
Oct 31 03:39:56.636585 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:39:56.636556    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25\": container with ID starting with 7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25 not found: ID does not exist" containerID="7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"
Oct 31 03:39:56.636663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.636595    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25"} err="failed to get container status \"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25\": rpc error: code = NotFound desc = could not find container \"7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25\": container with ID starting with 7dc1eb942dd161a9dfda43f9fb5f45843e57d5cfb88368818482b4645a7d5f25 not found: ID does not exist"
Oct 31 03:39:56.759012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.758978    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:39:56.762858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.762780    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-8g8hm"]
Oct 31 03:39:56.865851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.865814    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:39:56.868802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:56.868781    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-w48vf"]
Oct 31 03:39:58.136384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:58.136351    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" path="/var/lib/kubelet/pods/97432e9c-c79c-4bab-8cba-d9aa0eb475d3/volumes"
Oct 31 03:39:58.136867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:58.136852    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" path="/var/lib/kubelet/pods/d421a5c4-f284-4f4b-8904-41837c250ead/volumes"
Oct 31 03:39:58.583934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:39:58.583907    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:40:00.551182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:00.551148    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:40:01.559652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:01.559624    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:40:07.210187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.210146    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211050    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211075    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211089    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211097    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211115    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211124    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211138    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211146    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211174    5831 cpu_manager.go:414] "containerMap: removing container" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211182    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211328    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211344    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:07.216581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.211359    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="97432e9c-c79c-4bab-8cba-d9aa0eb475d3" containerName="istio-proxy"
Oct 31 03:40:07.222259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.222230    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.271848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.271718    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7a7f22-openshift-default-dockercfg-94w2x\""
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.313989    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314056    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314081    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6kp\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314109    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314137    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314175    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314300    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314421    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.315687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.314446    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.354146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.354102    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430110    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430173    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430315    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430428    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430452    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430565    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430598    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430621    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6kp\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.430900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.430647    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.432326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.432295    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.434224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.433042    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.437830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.437771    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.438909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.438856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.439904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.439860    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.452549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.450209    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.452549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.450439    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.518739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.518645    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6kp\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.520303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.520271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\") pod \"e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") " pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.557820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.557787    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:07.994515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:07.994458    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:40:07.997083 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:07.997051    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4f7fd6_8d21_44f2_be89_54ca99477da6.slice/crio-1efc2a12f96956618e772451adb92439ff074c33c325f3321d73f6235eea1c58 WatchSource:0}: Error finding container 1efc2a12f96956618e772451adb92439ff074c33c325f3321d73f6235eea1c58: Status 404 returned error can't find the container with id 1efc2a12f96956618e772451adb92439ff074c33c325f3321d73f6235eea1c58
Oct 31 03:40:08.000235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.000202    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:08.000344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.000285    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:08.000344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.000331    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:08.707144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.707105    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerStarted","Data":"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"}
Oct 31 03:40:08.707576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.707166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerStarted","Data":"1efc2a12f96956618e772451adb92439ff074c33c325f3321d73f6235eea1c58"}
Oct 31 03:40:08.817046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:08.816993    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podStartSLOduration=1.8169783179999999 podStartE2EDuration="1.816978318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:08.814071361 +0000 UTC m=+1958.828633068" watchObservedRunningTime="2026-10-31 03:40:08.816978318 +0000 UTC m=+1958.831540023"
Oct 31 03:40:09.558102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:09.558065    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:09.564342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:09.564317    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:09.716115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:09.716085    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:09.717088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:09.717069    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:40:12.979819 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:40:12.979778    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:40:15.313001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.312960    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:40:15.313743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.313724    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7181a37-6a28-4eb1-8005-05fe31a01474" containerName="istio-proxy"
Oct 31 03:40:15.313743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.313756    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d421a5c4-f284-4f4b-8904-41837c250ead" containerName="istio-proxy"
Oct 31 03:40:15.317541 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.317513    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.370259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.370225    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:40:15.420967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.420895    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7a7f22-dockercfg-2n94n\""
Oct 31 03:40:15.430787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.430750    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7a7f22\""
Oct 31 03:40:15.539506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.539472    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:40:15.541414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.541380    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6fd\" (UniqueName: \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.541839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.541798    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.543785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.543766    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:40:15.543916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.543847    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.643036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.642946    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.643195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.643118    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6fd\" (UniqueName: \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.643258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.643243    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx2b\" (UniqueName: \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.643312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.643293    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.644068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.644043    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.705187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.705155    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6fd\" (UniqueName: \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\") pod \"payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:15.744615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.744582    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.744776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.744711    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx2b\" (UniqueName: \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.745371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.745351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.798644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.798612    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx2b\" (UniqueName: \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\") pod \"payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.859979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.859943    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:15.933776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:15.932973    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:16.237342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.237312    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:40:16.239222 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:16.239198    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd0bc53_c7c9_4672_9ac7_549b3ab8d16b.slice/crio-0fb6f3d60bddc1bab3fe6dbd716340177c016f60829d17ca477fbd807bc78c3e WatchSource:0}: Error finding container 0fb6f3d60bddc1bab3fe6dbd716340177c016f60829d17ca477fbd807bc78c3e: Status 404 returned error can't find the container with id 0fb6f3d60bddc1bab3fe6dbd716340177c016f60829d17ca477fbd807bc78c3e
Oct 31 03:40:16.331499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.331473    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:40:16.333811 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:16.333784    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2585a498_2c01_4528_88f6_3e64d3c25dce.slice/crio-e292daeb025a2bb1756987af6a1df70535c4fac77fd159e72523d0e7d4390fe3 WatchSource:0}: Error finding container e292daeb025a2bb1756987af6a1df70535c4fac77fd159e72523d0e7d4390fe3: Status 404 returned error can't find the container with id e292daeb025a2bb1756987af6a1df70535c4fac77fd159e72523d0e7d4390fe3
Oct 31 03:40:16.793369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.793328    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" event={"ID":"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b","Type":"ContainerStarted","Data":"a2cb0fd10b37b93adf4fa3cd7f18f57b5dc61bcd534aa2599c19787419efc6aa"}
Oct 31 03:40:16.793640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.793576    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:16.793640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.793618    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" event={"ID":"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b","Type":"ContainerStarted","Data":"0fb6f3d60bddc1bab3fe6dbd716340177c016f60829d17ca477fbd807bc78c3e"}
Oct 31 03:40:16.795251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.795223    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" event={"ID":"2585a498-2c01-4528-88f6-3e64d3c25dce","Type":"ContainerStarted","Data":"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"}
Oct 31 03:40:16.795357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.795270    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" event={"ID":"2585a498-2c01-4528-88f6-3e64d3c25dce","Type":"ContainerStarted","Data":"e292daeb025a2bb1756987af6a1df70535c4fac77fd159e72523d0e7d4390fe3"}
Oct 31 03:40:16.795357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.795314    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:16.900181 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:16.900126    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" podStartSLOduration=1.9001110049999999 podStartE2EDuration="1.900111005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:16.898390811 +0000 UTC m=+1966.912952520" watchObservedRunningTime="2026-10-31 03:40:16.900111005 +0000 UTC m=+1966.914672760"
Oct 31 03:40:17.005967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:17.005898    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" podStartSLOduration=2.005879002 podStartE2EDuration="2.005879002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:17.003162538 +0000 UTC m=+1967.017724245" watchObservedRunningTime="2026-10-31 03:40:17.005879002 +0000 UTC m=+1967.020440712"
Oct 31 03:40:19.548095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.548057    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:40:19.552991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.552971    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.601445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.601415    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a32cdb\""
Oct 31 03:40:19.605248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.605216    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptvp\" (UniqueName: \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.605420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.605383    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.605705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.605680    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:40:19.647311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.647274    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a32cdb-dockercfg-t7dt4\""
Oct 31 03:40:19.707326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.707291    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.707565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.707432    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptvp\" (UniqueName: \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.708274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.708235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.760704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.760673    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptvp\" (UniqueName: \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\") pod \"payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:19.867818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:19.867734    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:20.247046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.247020    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:40:20.248895 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:20.248858    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e7a4ef_a65a_483b_b5be_ceffbf518e03.slice/crio-8bb10f71218e3db4ff332eb58cf22db33960fe10fbf48af2484bbf9b5942f0fb WatchSource:0}: Error finding container 8bb10f71218e3db4ff332eb58cf22db33960fe10fbf48af2484bbf9b5942f0fb: Status 404 returned error can't find the container with id 8bb10f71218e3db4ff332eb58cf22db33960fe10fbf48af2484bbf9b5942f0fb
Oct 31 03:40:20.281088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.281061    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:40:20.286591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.286570    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.315817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.315790    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.315978 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.315942    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffwj\" (UniqueName: \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.376684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.376644    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:40:20.417655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.417619    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffwj\" (UniqueName: \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.417868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.417773    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.418714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.418689    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.470934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.470898    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffwj\" (UniqueName: \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\") pod \"payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.626002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.625909    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:20.832085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.832049    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" event={"ID":"d2e7a4ef-a65a-483b-b5be-ceffbf518e03","Type":"ContainerStarted","Data":"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"}
Oct 31 03:40:20.832262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.832101    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" event={"ID":"d2e7a4ef-a65a-483b-b5be-ceffbf518e03","Type":"ContainerStarted","Data":"8bb10f71218e3db4ff332eb58cf22db33960fe10fbf48af2484bbf9b5942f0fb"}
Oct 31 03:40:20.832262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.832155    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:20.939789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:20.939743    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" podStartSLOduration=1.93972769 podStartE2EDuration="1.93972769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:20.937418368 +0000 UTC m=+1970.951980075" watchObservedRunningTime="2026-10-31 03:40:20.93972769 +0000 UTC m=+1970.954289476"
Oct 31 03:40:21.054574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:21.051936    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:40:21.055658 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:21.055615    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288aec47_ab90_45a2_a58e_90bccd98f0cd.slice/crio-59cd07a21fc6fda3acd8f27125c41550058bba5fb88bebb59fdf5b1bcb08a3e4 WatchSource:0}: Error finding container 59cd07a21fc6fda3acd8f27125c41550058bba5fb88bebb59fdf5b1bcb08a3e4: Status 404 returned error can't find the container with id 59cd07a21fc6fda3acd8f27125c41550058bba5fb88bebb59fdf5b1bcb08a3e4
Oct 31 03:40:21.840966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:21.840913    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" event={"ID":"288aec47-ab90-45a2-a58e-90bccd98f0cd","Type":"ContainerStarted","Data":"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"}
Oct 31 03:40:21.841484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:21.840979    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" event={"ID":"288aec47-ab90-45a2-a58e-90bccd98f0cd","Type":"ContainerStarted","Data":"59cd07a21fc6fda3acd8f27125c41550058bba5fb88bebb59fdf5b1bcb08a3e4"}
Oct 31 03:40:21.841484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:21.841196    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:21.950329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:21.950276    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" podStartSLOduration=1.950261862 podStartE2EDuration="1.950261862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:21.947448809 +0000 UTC m=+1971.962010516" watchObservedRunningTime="2026-10-31 03:40:21.950261862 +0000 UTC m=+1971.964823570"
Oct 31 03:40:23.622284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:23.622246    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:40:23.622801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:23.622473    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" containerID="cri-o://46c579f58f9783ff5ef6464e7864f53853ac35094c0e3019a26f21785770865f" gracePeriod=30
Oct 31 03:40:24.004454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:24.003391    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:40:24.004454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:24.003459    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:40:25.944238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:25.944198    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:40:25.950173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:25.950147    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.042404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.042371    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:40:26.044246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.044223    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-da5481-openshift-default-dockercfg-w4rr8\""
Oct 31 03:40:26.051928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.051890    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwplk\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.052257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.052219    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.052488 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.052470    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.052716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.052697    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.052915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.052881    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.053093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.053069    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.053288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.053271    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.053448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.053431    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.053682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.053656    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157641    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157730    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157770    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157808    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157863    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.157938    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwplk\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.158040    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.158256    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.158553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.158361    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.159198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.159098    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.169552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.169347    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.169730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.169654    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.170798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.169945    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.170965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.170897    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.172785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.172748    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.176332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.176245    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.222677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.222611    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.222807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.222775    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwplk\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\") pod \"e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") " pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.264176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.264143    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:26.687786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.687752    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:40:26.689247 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:26.689222    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead5ac1b_c6b4_4ea7_9bc2_8423134fa824.slice/crio-d7913c14a9eb99e831cfa574935c8c5cbce741a83640ab34ce34c3ef521eb0b9 WatchSource:0}: Error finding container d7913c14a9eb99e831cfa574935c8c5cbce741a83640ab34ce34c3ef521eb0b9: Status 404 returned error can't find the container with id d7913c14a9eb99e831cfa574935c8c5cbce741a83640ab34ce34c3ef521eb0b9
Oct 31 03:40:26.691841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.691808    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:26.691934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.691874    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:26.691934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.691902    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:26.884037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.884000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerStarted","Data":"c06d3a8a5174abb16e9358d3b7692256923aa024ae48b5b6fe6607769543477a"}
Oct 31 03:40:26.884205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:26.884053    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerStarted","Data":"d7913c14a9eb99e831cfa574935c8c5cbce741a83640ab34ce34c3ef521eb0b9"}
Oct 31 03:40:27.004452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.004344    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podStartSLOduration=2.004323035 podStartE2EDuration="2.004323035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:26.999902742 +0000 UTC m=+1977.014464449" watchObservedRunningTime="2026-10-31 03:40:27.004323035 +0000 UTC m=+1977.018884744"
Oct 31 03:40:27.264636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.264510    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:27.271507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.271479    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:27.809135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.809105    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:40:27.809308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.809184    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:40:27.893371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.893338    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:27.894366 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:27.894345    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:40:28.904101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904031    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerID="46c579f58f9783ff5ef6464e7864f53853ac35094c0e3019a26f21785770865f" exitCode=0
Oct 31 03:40:28.904557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904102    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerDied","Data":"46c579f58f9783ff5ef6464e7864f53853ac35094c0e3019a26f21785770865f"}
Oct 31 03:40:28.904557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904112    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:40:28.904557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904154    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq" event={"ID":"4ea37d7c-17bd-4875-b050-25f93945eed1","Type":"ContainerDied","Data":"345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e"}
Oct 31 03:40:28.904557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904174    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345922b1c7b5628ae25e5b902ac3e1fb75e1101bd8ead44ea58e31a97a90978e"
Oct 31 03:40:28.904557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:28.904208    5831 scope.go:122] "RemoveContainer" containerID="0c632bfc639b1a5dec6b455e3f3a1c6209a0bc465f30f10890df23bcfd30e331"
Oct 31 03:40:29.017496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017459    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.017698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017522    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.017698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017578    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.017698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017614    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.017698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017646    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.017905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017834    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:29.017977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017936    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:29.017977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.017963    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.018086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018001    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:29.018144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018106    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.018215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018169    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:29.018274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018244    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.018324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018272    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\") pod \"4ea37d7c-17bd-4875-b050-25f93945eed1\" (UID: \"4ea37d7c-17bd-4875-b050-25f93945eed1\") "
Oct 31 03:40:29.019383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018964    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.019383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.018987    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.019383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.019001    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.019383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.019015    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.019833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.019797    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:29.020603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.020576    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:29.020790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.020769    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:40:29.020964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.020944    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:29.021640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.021620    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm" pod "4ea37d7c-17bd-4875-b050-25f93945eed1" (UID: "4ea37d7c-17bd-4875-b050-25f93945eed1"). InnerVolumeSpecName "kube-api-access-n7lgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:29.120261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.120222    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lgm\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-kube-api-access-n7lgm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.120261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.120261    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.120486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.120274    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea37d7c-17bd-4875-b050-25f93945eed1-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.120486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.120286    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.120486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.120301    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea37d7c-17bd-4875-b050-25f93945eed1-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:29.916733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:29.916696    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"
Oct 31 03:40:30.178964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:30.178938    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:40:30.187167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:30.187138    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d3964-openshift-default-6646dbc887-t68nq"]
Oct 31 03:40:31.844459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:31.844430    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:40:32.137882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:32.137805    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" path="/var/lib/kubelet/pods/4ea37d7c-17bd-4875-b050-25f93945eed1/volumes"
Oct 31 03:40:32.852971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:32.852942    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:40:33.672733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.672690    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:40:33.673432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673409    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.673570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673433    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.673570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673454    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.673570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673461    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.673737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673699    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.673737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.673717    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea37d7c-17bd-4875-b050-25f93945eed1" containerName="istio-proxy"
Oct 31 03:40:33.678554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.678472    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.757165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.757126    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.780707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.780665    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgj6h\" (UniqueName: \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.814558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.791748    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-da5481-dockercfg-w7nsn\""
Oct 31 03:40:33.814558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.791817    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-da5481\""
Oct 31 03:40:33.850560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.839896    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:40:33.886175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.886146    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:40:33.890556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.890519    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:33.905618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.905596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.905754 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.905638    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj6h\" (UniqueName: \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.906511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.906484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:33.945289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.945254    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:40:33.959136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:33.959104    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj6h\" (UniqueName: \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\") pod \"payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:34.007239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.007203    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzbt\" (UniqueName: \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.007431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.007386    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.021524 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.021490    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:34.125070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.119747    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzbt\" (UniqueName: \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.125070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.120410    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.125070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.121716    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.173426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.173397    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzbt\" (UniqueName: \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\") pod \"payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.207313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.207235    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.417244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.417215    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:40:34.419169 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:34.419142    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616721c3_ae66_40b2_9f49_2c5059943ea6.slice/crio-0bb9ce2e66c0f48dfa936783fc3c3582f5752e385b35f33c2c4bc9f3d9c3f451 WatchSource:0}: Error finding container 0bb9ce2e66c0f48dfa936783fc3c3582f5752e385b35f33c2c4bc9f3d9c3f451: Status 404 returned error can't find the container with id 0bb9ce2e66c0f48dfa936783fc3c3582f5752e385b35f33c2c4bc9f3d9c3f451
Oct 31 03:40:34.585008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.584925    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:40:34.586404 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:34.586375    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfa7bcc_164a_4372_9557_a478b92b35f5.slice/crio-3276d26072c390a3193aecb2a7ebc9b0f74049f9d26ed059d0a5da33127d21bd WatchSource:0}: Error finding container 3276d26072c390a3193aecb2a7ebc9b0f74049f9d26ed059d0a5da33127d21bd: Status 404 returned error can't find the container with id 3276d26072c390a3193aecb2a7ebc9b0f74049f9d26ed059d0a5da33127d21bd
Oct 31 03:40:34.962618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.962575    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" event={"ID":"bbfa7bcc-164a-4372-9557-a478b92b35f5","Type":"ContainerStarted","Data":"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"}
Oct 31 03:40:34.963116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.962643    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" event={"ID":"bbfa7bcc-164a-4372-9557-a478b92b35f5","Type":"ContainerStarted","Data":"3276d26072c390a3193aecb2a7ebc9b0f74049f9d26ed059d0a5da33127d21bd"}
Oct 31 03:40:34.963116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.962762    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:34.964732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.964705    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" event={"ID":"616721c3-ae66-40b2-9f49-2c5059943ea6","Type":"ContainerStarted","Data":"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"}
Oct 31 03:40:34.964891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.964756    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" event={"ID":"616721c3-ae66-40b2-9f49-2c5059943ea6","Type":"ContainerStarted","Data":"0bb9ce2e66c0f48dfa936783fc3c3582f5752e385b35f33c2c4bc9f3d9c3f451"}
Oct 31 03:40:34.964891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.964782    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:34.972555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.972503    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:40:34.977361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:34.977338    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.161026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.160989    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c561e3-openshift-default-dockercfg-fkskp\""
Oct 31 03:40:35.161026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.161043    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:40:35.240033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.239892    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" podStartSLOduration=2.239872363 podStartE2EDuration="2.239872363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:35.234245938 +0000 UTC m=+1985.248807644" watchObservedRunningTime="2026-10-31 03:40:35.239872363 +0000 UTC m=+1985.254434072"
Oct 31 03:40:35.242982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.242954    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtnn\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243017    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243062    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243088    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243147    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243191    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243270    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243419    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.243501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.243486    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.341175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.341109    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" podStartSLOduration=2.341090483 podStartE2EDuration="2.341090483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:35.339471557 +0000 UTC m=+1985.354033282" watchObservedRunningTime="2026-10-31 03:40:35.341090483 +0000 UTC m=+1985.355652191"
Oct 31 03:40:35.344813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344789    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.344958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344839    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.344958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344879    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtnn\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.344958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344916    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344963    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.344988    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345030    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345080    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345140    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345257    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345396    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345602    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345609    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.345743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.345715    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.347509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.347490    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.347876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.347856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.397935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.397903    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.398084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.398065    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtnn\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\") pod \"e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") " pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.456022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.455722    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:35.941267 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:35.941214    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9627b327_9b81_4c03_87c2_f5f060fc0609.slice/crio-d5ed7b48e76567fc4c9b7ff14f9ea10b89eb4d2232edd99f28c43bb316746464 WatchSource:0}: Error finding container d5ed7b48e76567fc4c9b7ff14f9ea10b89eb4d2232edd99f28c43bb316746464: Status 404 returned error can't find the container with id d5ed7b48e76567fc4c9b7ff14f9ea10b89eb4d2232edd99f28c43bb316746464
Oct 31 03:40:35.941674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.941647    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:40:35.944234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.944199    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:35.944355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.944279    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:35.944355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.944320    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:35.997557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:35.997505    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerStarted","Data":"d5ed7b48e76567fc4c9b7ff14f9ea10b89eb4d2232edd99f28c43bb316746464"}
Oct 31 03:40:37.006607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:37.006573    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerStarted","Data":"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"}
Oct 31 03:40:37.119280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:37.119227    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podStartSLOduration=3.119213656 podStartE2EDuration="3.119213656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:37.115334218 +0000 UTC m=+1987.129895926" watchObservedRunningTime="2026-10-31 03:40:37.119213656 +0000 UTC m=+1987.133775495"
Oct 31 03:40:37.465808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:37.465773    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:37.471607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:37.471581    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:38.015319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:38.015280    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:38.016468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:38.016444    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:40:42.982495 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:40:42.982446    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:40:46.001201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:46.001168    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:40:46.001733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:46.001256    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:40:49.128437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.128387    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:40:49.128927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.128614    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" containerName="maas-api" containerID="cri-o://b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297" gracePeriod=30
Oct 31 03:40:49.475480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.475444    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:40:49.492909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.492881    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:40:49.623849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.623814    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\") pod \"fde8be45-364e-4575-8b04-3dc768de65e7\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") "
Oct 31 03:40:49.625236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.625211    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") pod \"fde8be45-364e-4575-8b04-3dc768de65e7\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") "
Oct 31 03:40:49.625548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.624438    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca" pod "fde8be45-364e-4575-8b04-3dc768de65e7" (UID: "fde8be45-364e-4575-8b04-3dc768de65e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:49.625718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.625469    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\") pod \"fde8be45-364e-4575-8b04-3dc768de65e7\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") "
Oct 31 03:40:49.626086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.626053    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\" (UniqueName: \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\") pod \"fde8be45-364e-4575-8b04-3dc768de65e7\" (UID: \"fde8be45-364e-4575-8b04-3dc768de65e7\") "
Oct 31 03:40:49.634744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.627800    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8be45-364e-4575-8b04-3dc768de65e7-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:49.634744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.629481    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs" pod "fde8be45-364e-4575-8b04-3dc768de65e7" (UID: "fde8be45-364e-4575-8b04-3dc768de65e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:49.641851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.641586    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj" pod "fde8be45-364e-4575-8b04-3dc768de65e7" (UID: "fde8be45-364e-4575-8b04-3dc768de65e7"). InnerVolumeSpecName "kube-api-access-4j9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:49.641851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.641584    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls" pod "fde8be45-364e-4575-8b04-3dc768de65e7" (UID: "fde8be45-364e-4575-8b04-3dc768de65e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:49.654088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.653966    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:40:49.654460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.654435    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" podUID="5cd3ec40-85c4-45bf-8326-00e7601bb237" containerName="payload-processing" containerID="cri-o://eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0" gracePeriod=30
Oct 31 03:40:49.711204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.711164    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:40:49.711452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.711407    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" podUID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" containerName="payload-pre-processing" containerID="cri-o://29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae" gracePeriod=30
Oct 31 03:40:49.729473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.729449    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:49.729473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.729475    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde8be45-364e-4575-8b04-3dc768de65e7-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:49.729640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.729487    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9mj\" (UniqueName: \"kubernetes.io/projected/fde8be45-364e-4575-8b04-3dc768de65e7-kube-api-access-4j9mj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:49.993771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:49.993730    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:40:50.033645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.033615    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:40:50.115084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.115044    5831 generic.go:356] "Generic (PLEG): container finished" podID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" containerID="29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae" exitCode=0
Oct 31 03:40:50.115298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.115125    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"
Oct 31 03:40:50.115298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.115169    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" event={"ID":"7ac0273d-8a7b-4d37-984e-6aedc278a6d3","Type":"ContainerDied","Data":"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"}
Oct 31 03:40:50.115298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.115233    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl" event={"ID":"7ac0273d-8a7b-4d37-984e-6aedc278a6d3","Type":"ContainerDied","Data":"fd9b4e68ea7768d06434cee5be57267c56ce9f46169b6a1dc629eea894b7f6ac"}
Oct 31 03:40:50.115298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.115263    5831 scope.go:122] "RemoveContainer" containerID="29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"
Oct 31 03:40:50.116922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.116895    5831 generic.go:356] "Generic (PLEG): container finished" podID="5cd3ec40-85c4-45bf-8326-00e7601bb237" containerID="eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0" exitCode=0
Oct 31 03:40:50.117096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.116936    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"
Oct 31 03:40:50.117096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.116988    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" event={"ID":"5cd3ec40-85c4-45bf-8326-00e7601bb237","Type":"ContainerDied","Data":"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"}
Oct 31 03:40:50.117096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.117027    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c" event={"ID":"5cd3ec40-85c4-45bf-8326-00e7601bb237","Type":"ContainerDied","Data":"a0085bf2ac7dbe3ce634133b63e7a492d3c90da933857905b987fd96a1b877e5"}
Oct 31 03:40:50.119074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.119049    5831 generic.go:356] "Generic (PLEG): container finished" podID="fde8be45-364e-4575-8b04-3dc768de65e7" containerID="b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297" exitCode=0
Oct 31 03:40:50.119207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.119088    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"
Oct 31 03:40:50.119207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.119172    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" event={"ID":"fde8be45-364e-4575-8b04-3dc768de65e7","Type":"ContainerDied","Data":"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"}
Oct 31 03:40:50.119322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.119210    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" event={"ID":"fde8be45-364e-4575-8b04-3dc768de65e7","Type":"ContainerDied","Data":"eebb8acd8f74cc4f50192771bdcfb3c5098619594ae3d64f82d7ebaf5feb7869"}
Oct 31 03:40:50.131606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.131339    5831 scope.go:122] "RemoveContainer" containerID="29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"
Oct 31 03:40:50.131993 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:40:50.131837    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae\": container with ID starting with 29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae not found: ID does not exist" containerID="29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"
Oct 31 03:40:50.131993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.131878    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae"} err="failed to get container status \"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae\": rpc error: code = NotFound desc = could not find container \"29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae\": container with ID starting with 29438ba67fa882aa1d82d0ca11bbbece3afdff22101fbfb9c8911cc5995949ae not found: ID does not exist"
Oct 31 03:40:50.131993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.131902    5831 scope.go:122] "RemoveContainer" containerID="eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"
Oct 31 03:40:50.134789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.134764    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\") pod \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") "
Oct 31 03:40:50.134937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.134826    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\" (UniqueName: \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\") pod \"5cd3ec40-85c4-45bf-8326-00e7601bb237\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") "
Oct 31 03:40:50.135033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.134968    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\") pod \"5cd3ec40-85c4-45bf-8326-00e7601bb237\" (UID: \"5cd3ec40-85c4-45bf-8326-00e7601bb237\") "
Oct 31 03:40:50.135133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.135043    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\" (UniqueName: \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\") pod \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\" (UID: \"7ac0273d-8a7b-4d37-984e-6aedc278a6d3\") "
Oct 31 03:40:50.135214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.135165    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume" pod "7ac0273d-8a7b-4d37-984e-6aedc278a6d3" (UID: "7ac0273d-8a7b-4d37-984e-6aedc278a6d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:50.135593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.135566    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:50.136025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.136001    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume" pod "5cd3ec40-85c4-45bf-8326-00e7601bb237" (UID: "5cd3ec40-85c4-45bf-8326-00e7601bb237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:50.138015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.137987    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj" pod "5cd3ec40-85c4-45bf-8326-00e7601bb237" (UID: "5cd3ec40-85c4-45bf-8326-00e7601bb237"). InnerVolumeSpecName "kube-api-access-v4lnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:50.138131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.138035    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh" pod "7ac0273d-8a7b-4d37-984e-6aedc278a6d3" (UID: "7ac0273d-8a7b-4d37-984e-6aedc278a6d3"). InnerVolumeSpecName "kube-api-access-p9gkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:50.147130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.147033    5831 scope.go:122] "RemoveContainer" containerID="eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"
Oct 31 03:40:50.147371 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:40:50.147339    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0\": container with ID starting with eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0 not found: ID does not exist" containerID="eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"
Oct 31 03:40:50.147494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.147375    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0"} err="failed to get container status \"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0\": rpc error: code = NotFound desc = could not find container \"eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0\": container with ID starting with eb81446e58470d1b5e587d1e85448590b497fbabdb43b9a20d5de5a0840e44d0 not found: ID does not exist"
Oct 31 03:40:50.147494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.147398    5831 scope.go:122] "RemoveContainer" containerID="b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"
Oct 31 03:40:50.167995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.167977    5831 scope.go:122] "RemoveContainer" containerID="b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"
Oct 31 03:40:50.168296 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:40:50.168277    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297\": container with ID starting with b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297 not found: ID does not exist" containerID="b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"
Oct 31 03:40:50.168375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.168301    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297"} err="failed to get container status \"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297\": rpc error: code = NotFound desc = could not find container \"b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297\": container with ID starting with b07106bf6c362c6e71e0a96503b28678471ab8d8d924095c99d4517a1ae9b297 not found: ID does not exist"
Oct 31 03:40:50.307547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.281258    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lnj\" (UniqueName: \"kubernetes.io/projected/5cd3ec40-85c4-45bf-8326-00e7601bb237-kube-api-access-v4lnj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:50.307547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.281293    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd3ec40-85c4-45bf-8326-00e7601bb237-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:50.307547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.281308    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9gkh\" (UniqueName: \"kubernetes.io/projected/7ac0273d-8a7b-4d37-984e-6aedc278a6d3-kube-api-access-p9gkh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:50.436158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.436127    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:40:50.439148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.439121    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8"]
Oct 31 03:40:50.587908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.587816    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:40:50.591678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.591641    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18d5f6-75c58d7fc9-64nnl"]
Oct 31 03:40:50.693910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.693872    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:40:50.698016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:50.697989    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18d5f6-59796c8dc6-hxt9c"]
Oct 31 03:40:52.136081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:52.136049    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd3ec40-85c4-45bf-8326-00e7601bb237" path="/var/lib/kubelet/pods/5cd3ec40-85c4-45bf-8326-00e7601bb237/volumes"
Oct 31 03:40:52.136494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:52.136434    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" path="/var/lib/kubelet/pods/7ac0273d-8a7b-4d37-984e-6aedc278a6d3/volumes"
Oct 31 03:40:52.136872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:52.136855    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" path="/var/lib/kubelet/pods/fde8be45-364e-4575-8b04-3dc768de65e7/volumes"
Oct 31 03:40:52.464605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:52.464490    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-18d5f6-5556d4dd7-k82h8" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.198:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:55.227765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:55.227729    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:40:55.228294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:55.227940    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy" containerID="cri-o://7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880" gracePeriod=30
Oct 31 03:40:57.296256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296224    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:40:57.296869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296849    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" containerName="maas-api"
Oct 31 03:40:57.296869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296868    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" containerName="maas-api"
Oct 31 03:40:57.297019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296906    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" containerName="payload-pre-processing"
Oct 31 03:40:57.297019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296911    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" containerName="payload-pre-processing"
Oct 31 03:40:57.297019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296933    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5cd3ec40-85c4-45bf-8326-00e7601bb237" containerName="payload-processing"
Oct 31 03:40:57.297019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.296938    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd3ec40-85c4-45bf-8326-00e7601bb237" containerName="payload-processing"
Oct 31 03:40:57.297019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.297015    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac0273d-8a7b-4d37-984e-6aedc278a6d3" containerName="payload-pre-processing"
Oct 31 03:40:57.297184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.297028    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd3ec40-85c4-45bf-8326-00e7601bb237" containerName="payload-processing"
Oct 31 03:40:57.297184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.297036    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde8be45-364e-4575-8b04-3dc768de65e7" containerName="maas-api"
Oct 31 03:40:57.302417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.302393    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.487681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.487647    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6f9fd2-openshift-default-dockercfg-5vdv2\""
Oct 31 03:40:57.490362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.488875    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqj7r\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.491624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.491601    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.491869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.491846    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.491999    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.492182    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.492376    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.492589    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.492793    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.492975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.492961    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.565715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.565614    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:40:57.604888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.604829    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqj7r\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.605193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.605167    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.605369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.605350    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.605517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.605500    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.605743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.605722    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.605963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.605944    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.606140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.606123    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.606316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.606297    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.606514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.606497    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.609594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.609523    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.609688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.609607    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.610309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.610231    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.610412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.610380    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.612553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.610696    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.615653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.613836    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.615653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.615398    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.663793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.663753    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqj7r\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.664000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.663980    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\") pod \"e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") " pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:57.921990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:57.921893    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:58.355651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:58.355611    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:40:58.392876 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:40:58.392847    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda052139c_cb90_4787_85b2_63bac51e0aed.slice/crio-d9ac467a4fe99b38736b17cfcc2c929e942964ca5b6bed650bceed4cc5fdda98 WatchSource:0}: Error finding container d9ac467a4fe99b38736b17cfcc2c929e942964ca5b6bed650bceed4cc5fdda98: Status 404 returned error can't find the container with id d9ac467a4fe99b38736b17cfcc2c929e942964ca5b6bed650bceed4cc5fdda98
Oct 31 03:40:58.395413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:58.395381    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:58.395512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:58.395452    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:58.395512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:58.395479    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:40:59.223122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:59.223080    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerStarted","Data":"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"}
Oct 31 03:40:59.223322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:59.223146    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerStarted","Data":"d9ac467a4fe99b38736b17cfcc2c929e942964ca5b6bed650bceed4cc5fdda98"}
Oct 31 03:40:59.378911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:59.378861    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podStartSLOduration=2.378845009 podStartE2EDuration="2.378845009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:59.376299845 +0000 UTC m=+2009.390861553" watchObservedRunningTime="2026-10-31 03:40:59.378845009 +0000 UTC m=+2009.393406721"
Oct 31 03:40:59.922830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:59.922791    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:40:59.929225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:40:59.929198    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:41:00.231960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.231930    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:41:00.233015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.232991    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:41:00.598447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.598418    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:41:00.665224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665192    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665245    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665281    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665341    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665386    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665407    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665455    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665484    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.665603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665495    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665606    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\") pod \"96154984-ffc5-48e8-a744-6032cddc2a00\" (UID: \"96154984-ffc5-48e8-a744-6032cddc2a00\") "
Oct 31 03:41:00.665878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665756    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.665878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:00.666037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665886    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.666037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.665969    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.666351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.666330    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.666481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.666354    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.666481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.666368    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96154984-ffc5-48e8-a744-6032cddc2a00-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.666481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.666380    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.666481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.666392    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.667934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.667908    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "kube-api-access-r89n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:00.668063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.668042    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.668347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.668328    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:41:00.668407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.668359    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token" pod "96154984-ffc5-48e8-a744-6032cddc2a00" (UID: "96154984-ffc5-48e8-a744-6032cddc2a00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:00.768147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.768109    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.768147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.768139    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96154984-ffc5-48e8-a744-6032cddc2a00-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.768147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.768154    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89n6\" (UniqueName: \"kubernetes.io/projected/96154984-ffc5-48e8-a744-6032cddc2a00-kube-api-access-r89n6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.768388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:00.768168    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96154984-ffc5-48e8-a744-6032cddc2a00-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:01.248838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.248750    5831 generic.go:356] "Generic (PLEG): container finished" podID="96154984-ffc5-48e8-a744-6032cddc2a00" containerID="7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880" exitCode=0
Oct 31 03:41:01.249019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.248806    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"
Oct 31 03:41:01.249104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.248806    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerDied","Data":"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"}
Oct 31 03:41:01.249167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.249144    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs" event={"ID":"96154984-ffc5-48e8-a744-6032cddc2a00","Type":"ContainerDied","Data":"a0c5f786372743d4ba1fe43a266c97fc978d7768f08bcb21405b8a0f2406ed57"}
Oct 31 03:41:01.249214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.249192    5831 scope.go:122] "RemoveContainer" containerID="7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"
Oct 31 03:41:01.263033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.263006    5831 scope.go:122] "RemoveContainer" containerID="bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"
Oct 31 03:41:01.273792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.273769    5831 scope.go:122] "RemoveContainer" containerID="7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"
Oct 31 03:41:01.274099 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:01.274081    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880\": container with ID starting with 7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880 not found: ID does not exist" containerID="7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"
Oct 31 03:41:01.274150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.274110    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880"} err="failed to get container status \"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880\": rpc error: code = NotFound desc = could not find container \"7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880\": container with ID starting with 7823c6b349f96974257410e6cf63d73aed5561f74d0dea0e3eab62b860e1d880 not found: ID does not exist"
Oct 31 03:41:01.274150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.274133    5831 scope.go:122] "RemoveContainer" containerID="bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"
Oct 31 03:41:01.274411 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:01.274389    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45\": container with ID starting with bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45 not found: ID does not exist" containerID="bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"
Oct 31 03:41:01.274463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.274421    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45"} err="failed to get container status \"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45\": rpc error: code = NotFound desc = could not find container \"bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45\": container with ID starting with bcf8c41624c10cf75e60edf684dd7028ce6df7c3a0450d0802a2c9f910940a45 not found: ID does not exist"
Oct 31 03:41:01.432181 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.432144    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:41:01.527245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:01.527145    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-18d5f6-openshift-default-5d69c6bd65-8m2xs"]
Oct 31 03:41:02.137371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:02.137340    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" path="/var/lib/kubelet/pods/96154984-ffc5-48e8-a744-6032cddc2a00/volumes"
Oct 31 03:41:07.122410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.122358    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:41:07.123196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.123175    5831 cpu_manager.go:414] "containerMap: removing container" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.123196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.123196    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.123359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.123306    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.128718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.128684    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.180487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.180454    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:41:07.226569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.226425    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6f9fd2\""
Oct 31 03:41:07.260142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.260111    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6f9fd2-dockercfg-f8w8q\""
Oct 31 03:41:07.342597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.320292    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.342597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.326582    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pmx8\" (UniqueName: \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.372092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.372051    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:41:07.373168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.373101    5831 cpu_manager.go:414] "containerMap: removing container" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.373168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.373132    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.373358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.373295    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="96154984-ffc5-48e8-a744-6032cddc2a00" containerName="istio-proxy"
Oct 31 03:41:07.377598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.377576    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:41:07.377716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.377689    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.427728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.427700    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.427904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.427883    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.427980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.427946    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqv2\" (UniqueName: \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.428074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.428057    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pmx8\" (UniqueName: \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.428627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.428608    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.480924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.480895    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pmx8\" (UniqueName: \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\") pod \"payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:07.528953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.528910    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqv2\" (UniqueName: \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.529154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.529084    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.529826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.529807    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.582046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.582010    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqv2\" (UniqueName: \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\") pod \"payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.692414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.692380    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:07.743759 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:07.743724    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:08.152355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.152322    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:41:08.154391 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:41:08.154364    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00df6b23_b876_401f_9f15_fa8fd6cc980a.slice/crio-f84b54050406fb5735b5424be50ce64299477449b2e7b2fd6f8f676f34cdbde1 WatchSource:0}: Error finding container f84b54050406fb5735b5424be50ce64299477449b2e7b2fd6f8f676f34cdbde1: Status 404 returned error can't find the container with id f84b54050406fb5735b5424be50ce64299477449b2e7b2fd6f8f676f34cdbde1
Oct 31 03:41:08.189979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.189949    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:41:08.197040 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:41:08.197017    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef5909c_2322_4be3_a4b2_d74a76aab0ad.slice/crio-73c3fe1b5aded78232d384477f7d0c11be7e3bd6f90c11b892068441da2b7736 WatchSource:0}: Error finding container 73c3fe1b5aded78232d384477f7d0c11be7e3bd6f90c11b892068441da2b7736: Status 404 returned error can't find the container with id 73c3fe1b5aded78232d384477f7d0c11be7e3bd6f90c11b892068441da2b7736
Oct 31 03:41:08.356130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.356087    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" event={"ID":"00df6b23-b876-401f-9f15-fa8fd6cc980a","Type":"ContainerStarted","Data":"6919f870ee674a6db41f1c2196a8a48a65ad1718f507f08a999e77d05c4e6775"}
Oct 31 03:41:08.356334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.356153    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" event={"ID":"00df6b23-b876-401f-9f15-fa8fd6cc980a","Type":"ContainerStarted","Data":"f84b54050406fb5735b5424be50ce64299477449b2e7b2fd6f8f676f34cdbde1"}
Oct 31 03:41:08.356334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.356181    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:08.357877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.357852    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" event={"ID":"1ef5909c-2322-4be3-a4b2-d74a76aab0ad","Type":"ContainerStarted","Data":"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"}
Oct 31 03:41:08.357996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.357900    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" event={"ID":"1ef5909c-2322-4be3-a4b2-d74a76aab0ad","Type":"ContainerStarted","Data":"73c3fe1b5aded78232d384477f7d0c11be7e3bd6f90c11b892068441da2b7736"}
Oct 31 03:41:08.357996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.357925    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:08.462490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.462432    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" podStartSLOduration=1.462417628 podStartE2EDuration="1.462417628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:08.460826328 +0000 UTC m=+2018.475388017" watchObservedRunningTime="2026-10-31 03:41:08.462417628 +0000 UTC m=+2018.476979341"
Oct 31 03:41:08.570076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:08.570018    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" podStartSLOduration=1.570004411 podStartE2EDuration="1.570004411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:08.565922582 +0000 UTC m=+2018.580484290" watchObservedRunningTime="2026-10-31 03:41:08.570004411 +0000 UTC m=+2018.584566119"
Oct 31 03:41:12.882832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.882796    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:41:12.887132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.887111    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:12.985590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.982035    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7a7f22-serving-cert\""
Oct 31 03:41:12.988385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.988341    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:41:12.994775 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:12.994735    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:41:12.997474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.997443    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2tp\" (UniqueName: \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:12.997684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.997560    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:12.997768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.997700    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:12.997870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:12.997852    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.099194 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.099147    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.099411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.099282    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2tp\" (UniqueName: \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.099411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.099349    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.099577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.099414    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.099669 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:13.099646    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-serving-cert: secret "maas-api-e2e-api-b-7a7f22-serving-cert" not found
Oct 31 03:41:13.099743 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:13.099731    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls podName:3e19927c-1a25-4bb0-87d4-5c92b8cbc614 nodeName:}" failed. No retries permitted until 2026-10-31 03:41:13.599710926 +0000 UTC m=+2023.614272616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls") pod "maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" (UID: "3e19927c-1a25-4bb0-87d4-5c92b8cbc614") : secret "maas-api-e2e-api-b-7a7f22-serving-cert" not found
Oct 31 03:41:13.100388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.100355    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.102210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.102183    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.153988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.153920    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2tp\" (UniqueName: \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.614468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.614433    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.617433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.617402    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") pod \"maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:13.803606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:13.803523    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:14.182457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:14.182429    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:41:14.184888 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:41:14.184855    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e19927c_1a25_4bb0_87d4_5c92b8cbc614.slice/crio-93fa073b7b2e123b0a46b8faf67d7f8bc4e2cc4b7f76f34fea36f33cd5c75e3e WatchSource:0}: Error finding container 93fa073b7b2e123b0a46b8faf67d7f8bc4e2cc4b7f76f34fea36f33cd5c75e3e: Status 404 returned error can't find the container with id 93fa073b7b2e123b0a46b8faf67d7f8bc4e2cc4b7f76f34fea36f33cd5c75e3e
Oct 31 03:41:14.408368 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:14.408332    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" event={"ID":"3e19927c-1a25-4bb0-87d4-5c92b8cbc614","Type":"ContainerStarted","Data":"93fa073b7b2e123b0a46b8faf67d7f8bc4e2cc4b7f76f34fea36f33cd5c75e3e"}
Oct 31 03:41:15.417482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:15.417388    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" event={"ID":"3e19927c-1a25-4bb0-87d4-5c92b8cbc614","Type":"ContainerStarted","Data":"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"}
Oct 31 03:41:15.417963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:15.417556    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:15.527614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:15.527556    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" podStartSLOduration=2.554567332 podStartE2EDuration="3.527524795s" totalImagesPullingTime="972.957463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:15.524512524 +0000 UTC m=+2025.539074232" watchObservedRunningTime="2026-10-31 03:41:15.527524795 +0000 UTC m=+2025.542086502"
Oct 31 03:41:19.368672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:19.368639    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:41:19.369083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:19.368783    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:41:21.431686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:21.431657    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:41:30.111552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:30.106514    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:41:34.762487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.762450    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:41:34.768862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.768840    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.820753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.820716    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:41:34.838299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.838269    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.838516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.838394    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2vb\" (UniqueName: \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.838516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.838444    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.838516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.838475    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.861724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.861672    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-da5481-serving-cert\""
Oct 31 03:41:34.939371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.939333    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.939562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.939402    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.939562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.939475    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.939648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.939633    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2vb\" (UniqueName: \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.939872 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:34.939781    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-serving-cert: secret "maas-api-e2e-praxis-a-da5481-serving-cert" not found
Oct 31 03:41:34.939872 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:34.939847    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls podName:4fec983c-d1d4-44f5-8980-0125058887ae nodeName:}" failed. No retries permitted until 2026-10-31 03:41:35.439825189 +0000 UTC m=+2045.454386889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls") pod "maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" (UID: "4fec983c-d1d4-44f5-8980-0125058887ae") : secret "maas-api-e2e-praxis-a-da5481-serving-cert" not found
Oct 31 03:41:34.940427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.940401    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:34.945571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:34.943882    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:35.010387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:35.010351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2vb\" (UniqueName: \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:35.447044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:35.447006    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:35.449652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:35.449627    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:35.692555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:35.690023    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:36.113918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:36.113878    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:41:36.114951 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:41:36.114925    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fec983c_d1d4_44f5_8980_0125058887ae.slice/crio-8f9b064c155093b2199f2d44303d8ee1394dc0d0db8e09a9d5b7e904c164e86c WatchSource:0}: Error finding container 8f9b064c155093b2199f2d44303d8ee1394dc0d0db8e09a9d5b7e904c164e86c: Status 404 returned error can't find the container with id 8f9b064c155093b2199f2d44303d8ee1394dc0d0db8e09a9d5b7e904c164e86c
Oct 31 03:41:36.585192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:36.585155    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" event={"ID":"4fec983c-d1d4-44f5-8980-0125058887ae","Type":"ContainerStarted","Data":"8f9b064c155093b2199f2d44303d8ee1394dc0d0db8e09a9d5b7e904c164e86c"}
Oct 31 03:41:37.599022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:37.598982    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" event={"ID":"4fec983c-d1d4-44f5-8980-0125058887ae","Type":"ContainerStarted","Data":"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"}
Oct 31 03:41:37.599572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:37.599089    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:37.707652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:37.707599    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" podStartSLOduration=2.81270187 podStartE2EDuration="3.707583838s" totalImagesPullingTime="894.881968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:37.705562181 +0000 UTC m=+2047.720123891" watchObservedRunningTime="2026-10-31 03:41:37.707583838 +0000 UTC m=+2047.722145573"
Oct 31 03:41:40.347505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.347470    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": read tcp 10.132.0.2:39050->10.132.0.204:15021: read: connection reset by peer" start-of-body=
Oct 31 03:41:40.347977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.347633    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": read tcp 10.132.0.2:39050->10.132.0.204:15021: read: connection reset by peer"
Oct 31 03:41:40.631752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.631641    5831 generic.go:356] "Generic (PLEG): container finished" podID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerID="706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff" exitCode=0
Oct 31 03:41:40.631752 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.631719    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerDied","Data":"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"}
Oct 31 03:41:40.632223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.632205    5831 scope.go:122] "RemoveContainer" containerID="706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"
Oct 31 03:41:40.633781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.633743    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:41:40.633871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.633827    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:41:40.633871 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:40.633863    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:41:41.643248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:41.643216    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerStarted","Data":"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"}
Oct 31 03:41:42.558771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:42.558730    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:41:42.558959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:42.558874    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 03:41:42.558959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:42.558919    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 03:41:42.996557 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:42.988758    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:41:43.558275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:43.558240    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 03:41:43.558510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:43.558292    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 03:41:43.621860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:43.621828    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:41:44.349450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.349416    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:41:44.350057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.349645    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" podUID="dcfdc610-fc63-4169-9721-9c7557bf0f51" containerName="maas-api" containerID="cri-o://00468f8327fad82eb8d8e044cea03395c8b41f9487c45f052b912300ef418f12" gracePeriod=30
Oct 31 03:41:44.558716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.558678    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 03:41:44.558886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.558742    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 03:41:44.754841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.754473    5831 generic.go:356] "Generic (PLEG): container finished" podID="dcfdc610-fc63-4169-9721-9c7557bf0f51" containerID="00468f8327fad82eb8d8e044cea03395c8b41f9487c45f052b912300ef418f12" exitCode=0
Oct 31 03:41:44.754841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.754739    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" event={"ID":"dcfdc610-fc63-4169-9721-9c7557bf0f51","Type":"ContainerDied","Data":"00468f8327fad82eb8d8e044cea03395c8b41f9487c45f052b912300ef418f12"}
Oct 31 03:41:44.879569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.872101    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:41:44.951209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.951174    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:41:44.951497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:44.951464    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" podUID="3f7eccd5-1425-4a0e-bf63-f87d50328670" containerName="payload-pre-processing" containerID="cri-o://7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701" gracePeriod=30
Oct 31 03:41:45.062124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.062042    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\") pod \"dcfdc610-fc63-4169-9721-9c7557bf0f51\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") "
Oct 31 03:41:45.062464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.062438    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") pod \"dcfdc610-fc63-4169-9721-9c7557bf0f51\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") "
Oct 31 03:41:45.062905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.062626    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\") pod \"dcfdc610-fc63-4169-9721-9c7557bf0f51\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") "
Oct 31 03:41:45.062905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.062841    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\" (UniqueName: \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\") pod \"dcfdc610-fc63-4169-9721-9c7557bf0f51\" (UID: \"dcfdc610-fc63-4169-9721-9c7557bf0f51\") "
Oct 31 03:41:45.063410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.063388    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca" pod "dcfdc610-fc63-4169-9721-9c7557bf0f51" (UID: "dcfdc610-fc63-4169-9721-9c7557bf0f51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:45.063912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.063894    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfdc610-fc63-4169-9721-9c7557bf0f51-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.068994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.068926    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls" pod "dcfdc610-fc63-4169-9721-9c7557bf0f51" (UID: "dcfdc610-fc63-4169-9721-9c7557bf0f51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:41:45.071411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.071382    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs" pod "dcfdc610-fc63-4169-9721-9c7557bf0f51" (UID: "dcfdc610-fc63-4169-9721-9c7557bf0f51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:41:45.071516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.071497    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd" pod "dcfdc610-fc63-4169-9721-9c7557bf0f51" (UID: "dcfdc610-fc63-4169-9721-9c7557bf0f51"). InnerVolumeSpecName "kube-api-access-nqmbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:45.165078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.165047    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.165078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.165079    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcfdc610-fc63-4169-9721-9c7557bf0f51-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.165311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.165090    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmbd\" (UniqueName: \"kubernetes.io/projected/dcfdc610-fc63-4169-9721-9c7557bf0f51-kube-api-access-nqmbd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.344918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.344888    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:41:45.489577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.483549    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\" (UniqueName: \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\") pod \"3f7eccd5-1425-4a0e-bf63-f87d50328670\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") "
Oct 31 03:41:45.489577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.483882    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\") pod \"3f7eccd5-1425-4a0e-bf63-f87d50328670\" (UID: \"3f7eccd5-1425-4a0e-bf63-f87d50328670\") "
Oct 31 03:41:45.489577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.484738    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume" pod "3f7eccd5-1425-4a0e-bf63-f87d50328670" (UID: "3f7eccd5-1425-4a0e-bf63-f87d50328670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:45.536555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.531331    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q" pod "3f7eccd5-1425-4a0e-bf63-f87d50328670" (UID: "3f7eccd5-1425-4a0e-bf63-f87d50328670"). InnerVolumeSpecName "kube-api-access-nkr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:45.559093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.559057    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 03:41:45.559409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.559365    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 03:41:45.586104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.585939    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eccd5-1425-4a0e-bf63-f87d50328670-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.586104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.585979    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkr5q\" (UniqueName: \"kubernetes.io/projected/3f7eccd5-1425-4a0e-bf63-f87d50328670-kube-api-access-nkr5q\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.764429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.764392    5831 generic.go:356] "Generic (PLEG): container finished" podID="3f7eccd5-1425-4a0e-bf63-f87d50328670" containerID="7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701" exitCode=0
Oct 31 03:41:45.764656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.764451    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"
Oct 31 03:41:45.764656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.764497    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" event={"ID":"3f7eccd5-1425-4a0e-bf63-f87d50328670","Type":"ContainerDied","Data":"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"}
Oct 31 03:41:45.764656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.764612    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99" event={"ID":"3f7eccd5-1425-4a0e-bf63-f87d50328670","Type":"ContainerDied","Data":"30a2c4858eb04e602dcc0928de222c8ecdce093648a5a54ad8c79a2810d1f31c"}
Oct 31 03:41:45.764656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.764643    5831 scope.go:122] "RemoveContainer" containerID="7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"
Oct 31 03:41:45.766561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.766483    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8" event={"ID":"dcfdc610-fc63-4169-9721-9c7557bf0f51","Type":"ContainerDied","Data":"12dd847ff7029f282844de405bf7da295bb5434543995832ba46a9aa538376c2"}
Oct 31 03:41:45.766561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.766521    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"
Oct 31 03:41:45.777964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.777947    5831 scope.go:122] "RemoveContainer" containerID="7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"
Oct 31 03:41:45.778257 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:45.778233    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701\": container with ID starting with 7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701 not found: ID does not exist" containerID="7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"
Oct 31 03:41:45.778353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.778262    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701"} err="failed to get container status \"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701\": rpc error: code = NotFound desc = could not find container \"7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701\": container with ID starting with 7afde9dbea9bc4260096121a4500bbedd1a00be8df8e94f2b6a26f713fe81701 not found: ID does not exist"
Oct 31 03:41:45.778353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.778279    5831 scope.go:122] "RemoveContainer" containerID="00468f8327fad82eb8d8e044cea03395c8b41f9487c45f052b912300ef418f12"
Oct 31 03:41:45.919736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.919608    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:41:45.922098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:45.922068    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b4030-554877f44c-j6g99"]
Oct 31 03:41:46.022600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.022565    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:41:46.026084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.026046    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b4030-589d78894c-wzlq8"]
Oct 31 03:41:46.122410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.122365    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:41:46.123307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123280    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dcfdc610-fc63-4169-9721-9c7557bf0f51" containerName="maas-api"
Oct 31 03:41:46.123307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123300    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfdc610-fc63-4169-9721-9c7557bf0f51" containerName="maas-api"
Oct 31 03:41:46.123515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123336    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3f7eccd5-1425-4a0e-bf63-f87d50328670" containerName="payload-pre-processing"
Oct 31 03:41:46.123515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123342    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7eccd5-1425-4a0e-bf63-f87d50328670" containerName="payload-pre-processing"
Oct 31 03:41:46.123515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123460    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfdc610-fc63-4169-9721-9c7557bf0f51" containerName="maas-api"
Oct 31 03:41:46.123515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.123479    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7eccd5-1425-4a0e-bf63-f87d50328670" containerName="payload-pre-processing"
Oct 31 03:41:46.128563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.128510    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.139494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.139464    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7eccd5-1425-4a0e-bf63-f87d50328670" path="/var/lib/kubelet/pods/3f7eccd5-1425-4a0e-bf63-f87d50328670/volumes"
Oct 31 03:41:46.139971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.139952    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfdc610-fc63-4169-9721-9c7557bf0f51" path="/var/lib/kubelet/pods/dcfdc610-fc63-4169-9721-9c7557bf0f51/volumes"
Oct 31 03:41:46.177441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.177366    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c561e3-serving-cert\""
Oct 31 03:41:46.203555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.200946    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:41:46.223747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.223628    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.224139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.224075    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.224280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.224122    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.224466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.224434    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvtt\" (UniqueName: \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.326849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.326810    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.327019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.326890    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.327019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.326932    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.327650 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:46.327291    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-serving-cert: secret "maas-api-e2e-ipp-b-c561e3-serving-cert" not found
Oct 31 03:41:46.327650 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:46.327372    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls podName:7909c7e4-b1a4-4476-a2f0-78bab221b44d nodeName:}" failed. No retries permitted until 2026-10-31 03:41:46.827349629 +0000 UTC m=+2056.841911338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls") pod "maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" (UID: "7909c7e4-b1a4-4476-a2f0-78bab221b44d") : secret "maas-api-e2e-ipp-b-c561e3-serving-cert" not found
Oct 31 03:41:46.328336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.328312    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.328494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.328361    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvtt\" (UniqueName: \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.335101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.335075    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.412556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.409256    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvtt\" (UniqueName: \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.561737 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.561702    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 03:41:46.562288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.561769    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 03:41:46.844033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.843929    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:46.846716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:46.846687    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:47.045296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.045265    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:47.424485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.424397    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:41:47.429264 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:41:47.429232    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7909c7e4_b1a4_4476_a2f0_78bab221b44d.slice/crio-e543d5aef92a60d86b2881acab5167ddafe44e64a3a9ed9ff84885b0755dbff2 WatchSource:0}: Error finding container e543d5aef92a60d86b2881acab5167ddafe44e64a3a9ed9ff84885b0755dbff2: Status 404 returned error can't find the container with id e543d5aef92a60d86b2881acab5167ddafe44e64a3a9ed9ff84885b0755dbff2
Oct 31 03:41:47.563476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.563447    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:41:47.563931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.563914    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:41:47.564637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.564615    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:41:47.795564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:47.795480    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" event={"ID":"7909c7e4-b1a4-4476-a2f0-78bab221b44d","Type":"ContainerStarted","Data":"e543d5aef92a60d86b2881acab5167ddafe44e64a3a9ed9ff84885b0755dbff2"}
Oct 31 03:41:48.807062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:48.807009    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" event={"ID":"7909c7e4-b1a4-4476-a2f0-78bab221b44d","Type":"ContainerStarted","Data":"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"}
Oct 31 03:41:48.807519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:48.807241    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:48.914680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:48.914614    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" podStartSLOduration=1.925696731 podStartE2EDuration="2.914595375s" totalImagesPullingTime="988.898644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:48.912614492 +0000 UTC m=+2058.927176205" watchObservedRunningTime="2026-10-31 03:41:48.914595375 +0000 UTC m=+2058.929157083"
Oct 31 03:41:52.324073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:52.324034    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:41:52.324917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:52.324881    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy" containerID="cri-o://11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193" gracePeriod=30
Oct 31 03:41:54.820425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:54.820390    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:41:57.668274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.668250    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:41:57.849314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849281    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849339    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849393    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849426    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849465    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849501    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849551    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849721    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849782    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\") pod \"b5467d59-dc1c-4d02-b4c7-436622d4f971\" (UID: \"b5467d59-dc1c-4d02-b4c7-436622d4f971\") "
Oct 31 03:41:57.849976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849786    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:57.849976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.849896    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:57.850190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850013    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:57.850255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850202    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:57.850623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850368    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.850623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850391    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467d59-dc1c-4d02-b4c7-436622d4f971-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.850623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850406    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.850623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850420    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.850623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.850580    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:57.853142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.853111    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:57.853142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.853119    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:41:57.853610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.853583    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:57.853732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.853690    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx" pod "b5467d59-dc1c-4d02-b4c7-436622d4f971" (UID: "b5467d59-dc1c-4d02-b4c7-436622d4f971"). InnerVolumeSpecName "kube-api-access-jsjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:57.879577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.879520    5831 generic.go:356] "Generic (PLEG): container finished" podID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerID="11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193" exitCode=0
Oct 31 03:41:57.879714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.879648    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"
Oct 31 03:41:57.879714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.879683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerDied","Data":"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"}
Oct 31 03:41:57.879850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.879751    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk" event={"ID":"b5467d59-dc1c-4d02-b4c7-436622d4f971","Type":"ContainerDied","Data":"9592dca161dc4d3997bea6796671a250809c307c645f20f4bff59bfcecbf5f2d"}
Oct 31 03:41:57.879850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.879779    5831 scope.go:122] "RemoveContainer" containerID="11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"
Oct 31 03:41:57.900888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.900862    5831 scope.go:122] "RemoveContainer" containerID="aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"
Oct 31 03:41:57.911670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.911647    5831 scope.go:122] "RemoveContainer" containerID="11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"
Oct 31 03:41:57.912143 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:57.912113    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193\": container with ID starting with 11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193 not found: ID does not exist" containerID="11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"
Oct 31 03:41:57.912248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.912155    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193"} err="failed to get container status \"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193\": rpc error: code = NotFound desc = could not find container \"11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193\": container with ID starting with 11d624867ea796224b5d854c80f07dbd82ecc8f6438ba63178b57d259d55f193 not found: ID does not exist"
Oct 31 03:41:57.912248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.912183    5831 scope.go:122] "RemoveContainer" containerID="aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"
Oct 31 03:41:57.912546 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:41:57.912504    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72\": container with ID starting with aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72 not found: ID does not exist" containerID="aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"
Oct 31 03:41:57.912644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.912557    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72"} err="failed to get container status \"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72\": rpc error: code = NotFound desc = could not find container \"aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72\": container with ID starting with aa20c8974d7dccfdb9ecc75d4ff8bba8b8925dda2c686cf85628958a747e6d72 not found: ID does not exist"
Oct 31 03:41:57.951719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.951688    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjkx\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-kube-api-access-jsjkx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.951719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.951719    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.951937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.951729    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.951937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.951739    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:57.951937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:57.951748    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467d59-dc1c-4d02-b4c7-436622d4f971-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:58.035316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:58.035278    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:41:58.038684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:58.038656    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8b4030-openshift-default-654d8bb565-tclsk"]
Oct 31 03:41:58.135908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:58.135875    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" path="/var/lib/kubelet/pods/b5467d59-dc1c-4d02-b4c7-436622d4f971/volumes"
Oct 31 03:41:58.934769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:58.934665    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:41:58.935160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:41:58.934761    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:01.993677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:01.993568    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerDied","Data":"c06d3a8a5174abb16e9358d3b7692256923aa024ae48b5b6fe6607769543477a"}
Oct 31 03:42:01.996288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:01.994892    5831 scope.go:122] "RemoveContainer" containerID="c06d3a8a5174abb16e9358d3b7692256923aa024ae48b5b6fe6607769543477a"
Oct 31 03:42:02.011555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:01.998202    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:02.011555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:01.998308    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:02.011555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:01.998353    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:02.026595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.020437    5831 generic.go:356] "Generic (PLEG): container finished" podID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerID="c06d3a8a5174abb16e9358d3b7692256923aa024ae48b5b6fe6607769543477a" exitCode=0
Oct 31 03:42:02.773453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.773411    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:42:02.774225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.774202    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:02.774225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.774225    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:02.774408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.774247    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:02.774408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.774255    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:02.774657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.774432    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:02.778184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.778162    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.824901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.824862    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.825095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.825004    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56pg\" (UniqueName: \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.827071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.827038    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c35264-dockercfg-lmrqq\""
Oct 31 03:42:02.827356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.827333    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c35264\""
Oct 31 03:42:02.831768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.831745    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:42:02.925801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.925763    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.925984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.925865    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56pg\" (UniqueName: \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.926453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.926433    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:02.978394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:02.978362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56pg\" (UniqueName: \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\") pod \"payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:03.031185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.031096    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerStarted","Data":"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"}
Oct 31 03:42:03.111930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.111894    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:03.265198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.265170    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:42:03.265392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.265360    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:03.265481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.265426    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:03.506347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.505905    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:42:03.507388 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:42:03.507345    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc533743e_ee72_4731_873a_e29da69e869c.slice/crio-9ca66d4b4377d18a835e7050e8e04d409405410e507520a3101649c75bc4c1a7 WatchSource:0}: Error finding container 9ca66d4b4377d18a835e7050e8e04d409405410e507520a3101649c75bc4c1a7: Status 404 returned error can't find the container with id 9ca66d4b4377d18a835e7050e8e04d409405410e507520a3101649c75bc4c1a7
Oct 31 03:42:03.555896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.555860    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:42:03.556799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.556765    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5467d59-dc1c-4d02-b4c7-436622d4f971" containerName="istio-proxy"
Oct 31 03:42:03.561588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.561568    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.614815 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.614783    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:42:03.636228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.636149    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcdf\" (UniqueName: \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.636386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.636270    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.738318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.738281    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcdf\" (UniqueName: \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.738318 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.738337    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.739096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.739075    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.808300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.808212    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcdf\" (UniqueName: \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\") pod \"payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:03.907557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:03.906987    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:04.042141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.042097    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" event={"ID":"c533743e-ee72-4731-873a-e29da69e869c","Type":"ContainerStarted","Data":"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"}
Oct 31 03:42:04.042659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.042166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" event={"ID":"c533743e-ee72-4731-873a-e29da69e869c","Type":"ContainerStarted","Data":"9ca66d4b4377d18a835e7050e8e04d409405410e507520a3101649c75bc4c1a7"}
Oct 31 03:42:04.042659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.042234    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:04.167337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.167238    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" podStartSLOduration=2.16722364 podStartE2EDuration="2.16722364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:04.16536522 +0000 UTC m=+2074.179926927" watchObservedRunningTime="2026-10-31 03:42:04.16722364 +0000 UTC m=+2074.181785347"
Oct 31 03:42:04.265351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.265312    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:04.265511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.265374    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:04.302218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:04.300474    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:42:04.349288 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:42:04.349254    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb365825_a2bc_4caf_aefd_958aeae6890b.slice/crio-f4003b40c144310f55ee9ba9e85c810e6878a0ecfafadd6a0e8fe4df4d004928 WatchSource:0}: Error finding container f4003b40c144310f55ee9ba9e85c810e6878a0ecfafadd6a0e8fe4df4d004928: Status 404 returned error can't find the container with id f4003b40c144310f55ee9ba9e85c810e6878a0ecfafadd6a0e8fe4df4d004928
Oct 31 03:42:05.052513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.052476    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" event={"ID":"bb365825-a2bc-4caf-aefd-958aeae6890b","Type":"ContainerStarted","Data":"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"}
Oct 31 03:42:05.053021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.052723    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:05.053021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.052755    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" event={"ID":"bb365825-a2bc-4caf-aefd-958aeae6890b","Type":"ContainerStarted","Data":"f4003b40c144310f55ee9ba9e85c810e6878a0ecfafadd6a0e8fe4df4d004928"}
Oct 31 03:42:05.162251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.162202    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" podStartSLOduration=2.162189787 podStartE2EDuration="2.162189787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:05.162139012 +0000 UTC m=+2075.176700720" watchObservedRunningTime="2026-10-31 03:42:05.162189787 +0000 UTC m=+2075.176751494"
Oct 31 03:42:05.265294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.265256    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:05.265470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:05.265315    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:06.265104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:06.265064    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:06.265613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:06.265136    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:07.264758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:07.264722    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:07.264964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:07.264783    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:08.264723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:08.264688    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 31 03:42:08.265241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:08.264747    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 31 03:42:09.269086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:09.269052    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:42:09.269620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:09.269511    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:42:09.270422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:09.270399    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:42:13.047565 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:42:13.046852    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:42:15.054463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:15.054431    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:42:15.586037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:15.585993    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:42:15.586263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:15.586232    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" containerID="cri-o://a68797c8f89fcec5c461f64d383afe8e5f4383df7e230e5115a94eb62554f3db" gracePeriod=30
Oct 31 03:42:16.101213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:16.094692    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:42:16.321223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:16.321192    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:42:20.349938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:20.349898    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:20.350374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:20.349996    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:21.241751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.241714    5831 generic.go:356] "Generic (PLEG): container finished" podID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerID="a68797c8f89fcec5c461f64d383afe8e5f4383df7e230e5115a94eb62554f3db" exitCode=0
Oct 31 03:42:21.242003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.241985    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" event={"ID":"9e95ce24-ffa0-4529-a5d7-4001b0f7365e","Type":"ContainerDied","Data":"a68797c8f89fcec5c461f64d383afe8e5f4383df7e230e5115a94eb62554f3db"}
Oct 31 03:42:21.348035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.348005    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:42:21.443486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443458    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443545    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443599    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443670    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443724    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443817    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443855    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.443946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.443899    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444019    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444183    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444217    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444289    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\") pod \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\" (UID: \"9e95ce24-ffa0-4529-a5d7-4001b0f7365e\") "
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444585    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444894    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444908    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444919    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444927    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.445305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.444936    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.446220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.446197    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:21.446466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.446435    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:21.446603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.446464    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "kube-api-access-78jqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:21.446660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.446636    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo" pod "9e95ce24-ffa0-4529-a5d7-4001b0f7365e" (UID: "9e95ce24-ffa0-4529-a5d7-4001b0f7365e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:21.546513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.546478    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.546513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.546512    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.546513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.546521    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:21.546785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:21.546554    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jqq\" (UniqueName: \"kubernetes.io/projected/9e95ce24-ffa0-4529-a5d7-4001b0f7365e-kube-api-access-78jqq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:22.250460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:22.250433    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"
Oct 31 03:42:22.250708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:22.250459    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc" event={"ID":"9e95ce24-ffa0-4529-a5d7-4001b0f7365e","Type":"ContainerDied","Data":"50eaf0e80d64d5564493276acbd37f180041b68a1252f3500ca088d7d62b37fd"}
Oct 31 03:42:22.250708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:22.250582    5831 scope.go:122] "RemoveContainer" containerID="a68797c8f89fcec5c461f64d383afe8e5f4383df7e230e5115a94eb62554f3db"
Oct 31 03:42:22.463504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:22.463471    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:42:22.466983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:22.466960    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-w5hkc"]
Oct 31 03:42:24.141603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:24.141564    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" path="/var/lib/kubelet/pods/9e95ce24-ffa0-4529-a5d7-4001b0f7365e/volumes"
Oct 31 03:42:27.310206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.310104    5831 generic.go:356] "Generic (PLEG): container finished" podID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerID="fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c" exitCode=0
Oct 31 03:42:27.310206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.310174    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerDied","Data":"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"}
Oct 31 03:42:27.310858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.310638    5831 scope.go:122] "RemoveContainer" containerID="fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"
Oct 31 03:42:27.316234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.312807    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:27.316234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.312986    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:27.316234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:27.313089    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:28.322008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:28.321971    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerStarted","Data":"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"}
Oct 31 03:42:28.465895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:28.465863    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:42:28.466108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:28.466084    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:28.466192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:28.466145    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:29.465725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:29.465687    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:29.466174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:29.465735    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:30.465949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:30.465913    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:30.466391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:30.465964    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:31.466453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:31.466419    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:31.466895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:31.466476    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:32.465611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:32.465577    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:32.465813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:32.465629    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:33.465779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:33.465735    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Oct 31 03:42:33.466189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:33.465795    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Oct 31 03:42:34.470465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:34.470435    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:42:34.470969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:34.470870    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:42:34.471639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:34.471619    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:42:38.141133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.141101    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:42:38.141639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.141101    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:42:38.156302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.156277    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:42:38.156302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.156286    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:42:38.160167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.160147    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:42:38.160313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:38.160220    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:42:42.982112 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:42:42.982071    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:42:45.233648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.233616    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:45.234082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.233680    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:45.609574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.609540    5831 generic.go:356] "Generic (PLEG): container finished" podID="a052139c-cb90-4787-85b2-63bac51e0aed" containerID="db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264" exitCode=0
Oct 31 03:42:45.609776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.609623    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerDied","Data":"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"}
Oct 31 03:42:45.610088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.610070    5831 scope.go:122] "RemoveContainer" containerID="db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"
Oct 31 03:42:45.612055 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.612020    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:45.612187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.612115    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:45.612187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:45.612163    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:42:46.624708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:46.624665    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerStarted","Data":"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"}
Oct 31 03:42:46.922396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:46.922308    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:42:46.922665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:46.922602    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:46.922756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:46.922677    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:47.922442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:47.922404    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:47.922903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:47.922474    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:48.923164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:48.923121    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:48.923648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:48.923208    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:49.922632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:49.922593    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:49.922826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:49.922661    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:50.923365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:50.923319    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 03:42:50.923925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:50.923390    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 03:42:51.984294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:51.984258    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:42:51.984859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:51.984841    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:42:51.985827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:51.985801    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:42:55.776115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:42:55.776080    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:43:07.757431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:07.757395    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:43:07.757986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:07.757638    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" podUID="ba3d7995-7216-478e-8244-e0bcab6f5145" containerName="maas-api" containerID="cri-o://081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d" gracePeriod=30
Oct 31 03:43:08.108288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.108257    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:43:08.218074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218040    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\" (UniqueName: \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\") pod \"ba3d7995-7216-478e-8244-e0bcab6f5145\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") "
Oct 31 03:43:08.218294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218092    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\") pod \"ba3d7995-7216-478e-8244-e0bcab6f5145\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") "
Oct 31 03:43:08.218294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218125    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\") pod \"ba3d7995-7216-478e-8244-e0bcab6f5145\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") "
Oct 31 03:43:08.218294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218231    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") pod \"ba3d7995-7216-478e-8244-e0bcab6f5145\" (UID: \"ba3d7995-7216-478e-8244-e0bcab6f5145\") "
Oct 31 03:43:08.218484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218458    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca" pod "ba3d7995-7216-478e-8244-e0bcab6f5145" (UID: "ba3d7995-7216-478e-8244-e0bcab6f5145"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:08.218789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.218771    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3d7995-7216-478e-8244-e0bcab6f5145-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.220498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.220472    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls" pod "ba3d7995-7216-478e-8244-e0bcab6f5145" (UID: "ba3d7995-7216-478e-8244-e0bcab6f5145"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:08.220601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.220491    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs" pod "ba3d7995-7216-478e-8244-e0bcab6f5145" (UID: "ba3d7995-7216-478e-8244-e0bcab6f5145"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:08.221691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.221668    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn" pod "ba3d7995-7216-478e-8244-e0bcab6f5145" (UID: "ba3d7995-7216-478e-8244-e0bcab6f5145"). InnerVolumeSpecName "kube-api-access-6z7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:08.248739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.248700    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:43:08.249033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.248981    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" podUID="bb365825-a2bc-4caf-aefd-958aeae6890b" containerName="payload-processing" containerID="cri-o://9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466" gracePeriod=30
Oct 31 03:43:08.304417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.304389    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:43:08.304699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.304677    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" podUID="c533743e-ee72-4731-873a-e29da69e869c" containerName="payload-pre-processing" containerID="cri-o://30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b" gracePeriod=30
Oct 31 03:43:08.319924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.319866    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7wn\" (UniqueName: \"kubernetes.io/projected/ba3d7995-7216-478e-8244-e0bcab6f5145-kube-api-access-6z7wn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.319924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.319890    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.319924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.319899    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba3d7995-7216-478e-8244-e0bcab6f5145-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.580868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.580839    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:43:08.613237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.613212    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:43:08.726742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.726701    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\" (UniqueName: \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\") pod \"c533743e-ee72-4731-873a-e29da69e869c\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") "
Oct 31 03:43:08.726959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.726818    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\" (UniqueName: \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\") pod \"bb365825-a2bc-4caf-aefd-958aeae6890b\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") "
Oct 31 03:43:08.726959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.726854    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\") pod \"bb365825-a2bc-4caf-aefd-958aeae6890b\" (UID: \"bb365825-a2bc-4caf-aefd-958aeae6890b\") "
Oct 31 03:43:08.727121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.727008    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\") pod \"c533743e-ee72-4731-873a-e29da69e869c\" (UID: \"c533743e-ee72-4731-873a-e29da69e869c\") "
Oct 31 03:43:08.728378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.728318    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume" pod "c533743e-ee72-4731-873a-e29da69e869c" (UID: "c533743e-ee72-4731-873a-e29da69e869c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:08.729063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.728954    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume" pod "bb365825-a2bc-4caf-aefd-958aeae6890b" (UID: "bb365825-a2bc-4caf-aefd-958aeae6890b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:08.730010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.729980    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg" pod "c533743e-ee72-4731-873a-e29da69e869c" (UID: "c533743e-ee72-4731-873a-e29da69e869c"). InnerVolumeSpecName "kube-api-access-s56pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:08.730165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.730143    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf" pod "bb365825-a2bc-4caf-aefd-958aeae6890b" (UID: "bb365825-a2bc-4caf-aefd-958aeae6890b"). InnerVolumeSpecName "kube-api-access-4fcdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:08.856563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.856219    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcdf\" (UniqueName: \"kubernetes.io/projected/bb365825-a2bc-4caf-aefd-958aeae6890b-kube-api-access-4fcdf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.856563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.856258    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb365825-a2bc-4caf-aefd-958aeae6890b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.856563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.856275    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c533743e-ee72-4731-873a-e29da69e869c-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.856563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.856292    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56pg\" (UniqueName: \"kubernetes.io/projected/c533743e-ee72-4731-873a-e29da69e869c-kube-api-access-s56pg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:08.888152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.888087    5831 generic.go:356] "Generic (PLEG): container finished" podID="ba3d7995-7216-478e-8244-e0bcab6f5145" containerID="081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d" exitCode=0
Oct 31 03:43:08.888382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.888239    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" event={"ID":"ba3d7995-7216-478e-8244-e0bcab6f5145","Type":"ContainerDied","Data":"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"}
Oct 31 03:43:08.888382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.888290    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx" event={"ID":"ba3d7995-7216-478e-8244-e0bcab6f5145","Type":"ContainerDied","Data":"608bc5679f2fc46ac97848a9d9af6aa94393e05bd2f6b76dbd8bf948d06acc03"}
Oct 31 03:43:08.888382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.888318    5831 scope.go:122] "RemoveContainer" containerID="081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"
Oct 31 03:43:08.888600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.888485    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"
Oct 31 03:43:08.892851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.892583    5831 generic.go:356] "Generic (PLEG): container finished" podID="c533743e-ee72-4731-873a-e29da69e869c" containerID="30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b" exitCode=0
Oct 31 03:43:08.892851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.892794    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"
Oct 31 03:43:08.894590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.894561    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" event={"ID":"c533743e-ee72-4731-873a-e29da69e869c","Type":"ContainerDied","Data":"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"}
Oct 31 03:43:08.894762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.894747    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6" event={"ID":"c533743e-ee72-4731-873a-e29da69e869c","Type":"ContainerDied","Data":"9ca66d4b4377d18a835e7050e8e04d409405410e507520a3101649c75bc4c1a7"}
Oct 31 03:43:08.899433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.896987    5831 generic.go:356] "Generic (PLEG): container finished" podID="bb365825-a2bc-4caf-aefd-958aeae6890b" containerID="9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466" exitCode=0
Oct 31 03:43:08.899433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.897062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" event={"ID":"bb365825-a2bc-4caf-aefd-958aeae6890b","Type":"ContainerDied","Data":"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"}
Oct 31 03:43:08.899433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.897101    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc" event={"ID":"bb365825-a2bc-4caf-aefd-958aeae6890b","Type":"ContainerDied","Data":"f4003b40c144310f55ee9ba9e85c810e6878a0ecfafadd6a0e8fe4df4d004928"}
Oct 31 03:43:08.899433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.897167    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"
Oct 31 03:43:08.916330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.916303    5831 scope.go:122] "RemoveContainer" containerID="081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"
Oct 31 03:43:08.918950 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:08.918920    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d\": container with ID starting with 081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d not found: ID does not exist" containerID="081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"
Oct 31 03:43:08.919065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.918960    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d"} err="failed to get container status \"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d\": rpc error: code = NotFound desc = could not find container \"081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d\": container with ID starting with 081bd00293b650b984057a5d8e04a7efadf4a33ec131a00ba9f0e321a940050d not found: ID does not exist"
Oct 31 03:43:08.919065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.918982    5831 scope.go:122] "RemoveContainer" containerID="30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"
Oct 31 03:43:08.930616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.930597    5831 scope.go:122] "RemoveContainer" containerID="30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"
Oct 31 03:43:08.930928 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:08.930907    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b\": container with ID starting with 30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b not found: ID does not exist" containerID="30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"
Oct 31 03:43:08.930986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.930935    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b"} err="failed to get container status \"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b\": rpc error: code = NotFound desc = could not find container \"30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b\": container with ID starting with 30c8d6ecd93917a2f6ed96a01c838562a461e0c358fe2284b02416582fd07c2b not found: ID does not exist"
Oct 31 03:43:08.930986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.930953    5831 scope.go:122] "RemoveContainer" containerID="9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"
Oct 31 03:43:08.942624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.942605    5831 scope.go:122] "RemoveContainer" containerID="9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"
Oct 31 03:43:08.942914 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:08.942895    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466\": container with ID starting with 9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466 not found: ID does not exist" containerID="9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"
Oct 31 03:43:08.942994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:08.942924    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466"} err="failed to get container status \"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466\": rpc error: code = NotFound desc = could not find container \"9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466\": container with ID starting with 9a7d72e689879fb568606e7b336f66ba0d5edfb82a548a93278ae644eec3b466 not found: ID does not exist"
Oct 31 03:43:09.045048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.044939    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:43:09.053570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.050479    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c35264-69c957b9b6-llmxx"]
Oct 31 03:43:09.153640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.153602    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:43:09.163917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.163890    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-5f6b77cf46-zgnmc"]
Oct 31 03:43:09.273155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.273123    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:43:09.275805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:09.275784    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-6df6cb7487-jkkf6"]
Oct 31 03:43:10.136347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:10.136312    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3d7995-7216-478e-8244-e0bcab6f5145" path="/var/lib/kubelet/pods/ba3d7995-7216-478e-8244-e0bcab6f5145/volumes"
Oct 31 03:43:10.136953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:10.136932    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb365825-a2bc-4caf-aefd-958aeae6890b" path="/var/lib/kubelet/pods/bb365825-a2bc-4caf-aefd-958aeae6890b/volumes"
Oct 31 03:43:10.137345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:10.137326    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c533743e-ee72-4731-873a-e29da69e869c" path="/var/lib/kubelet/pods/c533743e-ee72-4731-873a-e29da69e869c/volumes"
Oct 31 03:43:12.979075 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:12.979033    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:43:14.437126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:14.437094    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:43:14.437629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:14.437343    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy" containerID="cri-o://51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f" gracePeriod=30
Oct 31 03:43:19.752255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.752222    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:43:19.866743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.866713    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.866938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.866814    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.866938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.866870    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.866938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.866919    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.866962    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867010    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867027    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:19.867296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867143    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867186    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867203    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:19.867296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867248    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\") pod \"f5453eab-bd30-4ec8-93d9-5f8660860e42\" (UID: \"f5453eab-bd30-4ec8-93d9-5f8660860e42\") "
Oct 31 03:43:19.867296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:19.867594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867435    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:19.867594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.867482    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:19.868080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.868056    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.868189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.868085    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.868189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.868101    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.868189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.868120    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.868189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.868134    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5453eab-bd30-4ec8-93d9-5f8660860e42-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.869583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.869521    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:19.869709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.869595    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:19.869709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.869609    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "kube-api-access-bfxqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:19.869709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.869660    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo" pod "f5453eab-bd30-4ec8-93d9-5f8660860e42" (UID: "f5453eab-bd30-4ec8-93d9-5f8660860e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:19.969273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.969180    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.969273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.969211    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxqf\" (UniqueName: \"kubernetes.io/projected/f5453eab-bd30-4ec8-93d9-5f8660860e42-kube-api-access-bfxqf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.969273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.969228    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.969273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.969243    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5453eab-bd30-4ec8-93d9-5f8660860e42-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:19.989463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.989429    5831 generic.go:356] "Generic (PLEG): container finished" podID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerID="51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f" exitCode=0
Oct 31 03:43:19.989697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.989485    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"
Oct 31 03:43:19.989697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.989509    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerDied","Data":"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"}
Oct 31 03:43:19.989697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.989607    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79" event={"ID":"f5453eab-bd30-4ec8-93d9-5f8660860e42","Type":"ContainerDied","Data":"c4da920b43baf934ae8bead9fd184a723c96a3349c2f67b907ef80f864ec2458"}
Oct 31 03:43:19.989697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:19.989642    5831 scope.go:122] "RemoveContainer" containerID="51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"
Oct 31 03:43:20.003306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.003282    5831 scope.go:122] "RemoveContainer" containerID="61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"
Oct 31 03:43:20.013289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.013269    5831 scope.go:122] "RemoveContainer" containerID="51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"
Oct 31 03:43:20.013620 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:20.013596    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f\": container with ID starting with 51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f not found: ID does not exist" containerID="51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"
Oct 31 03:43:20.013687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.013629    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f"} err="failed to get container status \"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f\": rpc error: code = NotFound desc = could not find container \"51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f\": container with ID starting with 51902e24100be8f573b15b9bd3e48e70920c344136918d08ea2e39b71728b70f not found: ID does not exist"
Oct 31 03:43:20.013687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.013647    5831 scope.go:122] "RemoveContainer" containerID="61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"
Oct 31 03:43:20.013943 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:20.013919    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8\": container with ID starting with 61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8 not found: ID does not exist" containerID="61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"
Oct 31 03:43:20.013986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.013955    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8"} err="failed to get container status \"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8\": rpc error: code = NotFound desc = could not find container \"61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8\": container with ID starting with 61684b5115b6a576178a47cf61942491a568d47ec14b851294decbb9ebfba0a8 not found: ID does not exist"
Oct 31 03:43:20.151436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.151405    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:43:20.151908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.151880    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-329d79-openshift-default-575867c7-zmd79"]
Oct 31 03:43:20.288030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.287936    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:43:20.288919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288898    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:20.288990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288923    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:20.288990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288936    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c533743e-ee72-4731-873a-e29da69e869c" containerName="payload-pre-processing"
Oct 31 03:43:20.288990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288943    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c533743e-ee72-4731-873a-e29da69e869c" containerName="payload-pre-processing"
Oct 31 03:43:20.288990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288984    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ba3d7995-7216-478e-8244-e0bcab6f5145" containerName="maas-api"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.288991    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3d7995-7216-478e-8244-e0bcab6f5145" containerName="maas-api"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289019    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289026    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289047    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289053    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289067    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bb365825-a2bc-4caf-aefd-958aeae6890b" containerName="payload-processing"
Oct 31 03:43:20.289122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289074    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb365825-a2bc-4caf-aefd-958aeae6890b" containerName="payload-processing"
Oct 31 03:43:20.289344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289218    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:20.289344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289230    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c533743e-ee72-4731-873a-e29da69e869c" containerName="payload-pre-processing"
Oct 31 03:43:20.289344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289245    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3d7995-7216-478e-8244-e0bcab6f5145" containerName="maas-api"
Oct 31 03:43:20.289344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289258    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e95ce24-ffa0-4529-a5d7-4001b0f7365e" containerName="istio-proxy"
Oct 31 03:43:20.289344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.289267    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb365825-a2bc-4caf-aefd-958aeae6890b" containerName="payload-processing"
Oct 31 03:43:20.295229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.295205    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.363756    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.365007    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-dab9ee-openshift-default-dockercfg-zc7k5\""
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.374870    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.374928    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.374983    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375084    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375112    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwxt\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375151    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375258    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375346    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.391571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.375383    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476679    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476697    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwxt\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476718    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.476850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476850    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476874    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476912    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.476948    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.477131    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.477195    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.477289    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.477370    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.477854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.477829    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.479243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.479220    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.479843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.479821    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.563598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.560585    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.565914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.565778    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwxt\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\") pod \"e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") " pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:20.612787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:20.612752    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:21.072152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:21.072120    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:43:21.073641 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:21.073613    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bf809b_e77c_4145_8949_b4935befb5e7.slice/crio-2841113e917a2955dad87d8506c49958da5f7edc53caa6f26f8b5f0f2367f21f WatchSource:0}: Error finding container 2841113e917a2955dad87d8506c49958da5f7edc53caa6f26f8b5f0f2367f21f: Status 404 returned error can't find the container with id 2841113e917a2955dad87d8506c49958da5f7edc53caa6f26f8b5f0f2367f21f
Oct 31 03:43:21.076584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:21.076517    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:21.076693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:21.076621    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:21.076693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:21.076652    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:22.012475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.012438    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerStarted","Data":"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"}
Oct 31 03:43:22.012751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.012496    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerStarted","Data":"2841113e917a2955dad87d8506c49958da5f7edc53caa6f26f8b5f0f2367f21f"}
Oct 31 03:43:22.137121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.137087    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" path="/var/lib/kubelet/pods/f5453eab-bd30-4ec8-93d9-5f8660860e42/volumes"
Oct 31 03:43:22.183260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.183191    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podStartSLOduration=2.183170854 podStartE2EDuration="2.183170854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:22.181071998 +0000 UTC m=+2152.195633710" watchObservedRunningTime="2026-10-31 03:43:22.183170854 +0000 UTC m=+2152.197732563"
Oct 31 03:43:22.296441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.296330    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:43:22.297626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.297517    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5453eab-bd30-4ec8-93d9-5f8660860e42" containerName="istio-proxy"
Oct 31 03:43:22.307143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.307105    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.355814 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.355649    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.379245    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d22078-openshift-default-dockercfg-c9kqg\""
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415687    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415755    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415840    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415877    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415910    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.415970    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.416006    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwl8\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.416063    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.435448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.416135    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.517724    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.517837    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwl8\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.518555    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.518648    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.518823    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.518936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.518887    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.519115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.518986    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.519115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.519045    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.519115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.519081    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.519275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.519257    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.535619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.519560    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.535619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.519770    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.535619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.520011    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.535619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.520098    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.540477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.540334    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.540729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.540632    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.590864 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.590305    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwl8\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.621360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.621276    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:22.622567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.622472    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\") pod \"e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") " pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.647764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.645435    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:22.647764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.647092    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:22.923685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:22.923495    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:43:23.023192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.023161    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:23.024313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.024287    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:43:23.035641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.035609    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:43:23.123886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.122707    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:43:23.126129 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:23.126095    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7f4dc9_7ab8_4cdf_aa20_0cd3402fd48a.slice/crio-2d93bc9458e07677322a447a56a796ddcf93040817df65b4d36ee191b845a7bf WatchSource:0}: Error finding container 2d93bc9458e07677322a447a56a796ddcf93040817df65b4d36ee191b845a7bf: Status 404 returned error can't find the container with id 2d93bc9458e07677322a447a56a796ddcf93040817df65b4d36ee191b845a7bf
Oct 31 03:43:23.129484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.129445    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:23.129626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.129574    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:23.129626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:23.129615    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:24.033359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:24.033310    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerStarted","Data":"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"}
Oct 31 03:43:24.033818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:24.033420    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerStarted","Data":"2d93bc9458e07677322a447a56a796ddcf93040817df65b4d36ee191b845a7bf"}
Oct 31 03:43:24.154291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:24.154241    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podStartSLOduration=2.154227444 podStartE2EDuration="2.154227444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:24.152993548 +0000 UTC m=+2154.167555256" watchObservedRunningTime="2026-10-31 03:43:24.154227444 +0000 UTC m=+2154.168789151"
Oct 31 03:43:24.647604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:24.647569    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:24.654270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:24.654242    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:25.042481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:25.042442    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:25.043634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:25.043610    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:43:26.606947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:26.606914    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:43:26.607387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:26.607128    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" podUID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" containerName="maas-api" containerID="cri-o://5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c" gracePeriod=30
Oct 31 03:43:26.931989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:26.931961    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:43:27.062180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.062139    5831 generic.go:356] "Generic (PLEG): container finished" podID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" containerID="5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c" exitCode=0
Oct 31 03:43:27.062396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.062191    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"
Oct 31 03:43:27.062396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.062213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" event={"ID":"3e19927c-1a25-4bb0-87d4-5c92b8cbc614","Type":"ContainerDied","Data":"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"}
Oct 31 03:43:27.062396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.062279    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k" event={"ID":"3e19927c-1a25-4bb0-87d4-5c92b8cbc614","Type":"ContainerDied","Data":"93fa073b7b2e123b0a46b8faf67d7f8bc4e2cc4b7f76f34fea36f33cd5c75e3e"}
Oct 31 03:43:27.062396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.062310    5831 scope.go:122] "RemoveContainer" containerID="5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"
Oct 31 03:43:27.077135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.077112    5831 scope.go:122] "RemoveContainer" containerID="5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"
Oct 31 03:43:27.077482 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:27.077452    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c\": container with ID starting with 5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c not found: ID does not exist" containerID="5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"
Oct 31 03:43:27.077585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.077494    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c"} err="failed to get container status \"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c\": rpc error: code = NotFound desc = could not find container \"5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c\": container with ID starting with 5205c13fb02c4d14b243b5865cdbcb2f85ebcfb47f6601b4b13503fed8aa4e8c not found: ID does not exist"
Oct 31 03:43:27.104162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104129    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\") pod \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") "
Oct 31 03:43:27.104314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104176    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") pod \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") "
Oct 31 03:43:27.104314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104221    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\" (UniqueName: \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\") pod \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") "
Oct 31 03:43:27.104431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104312    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\") pod \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\" (UID: \"3e19927c-1a25-4bb0-87d4-5c92b8cbc614\") "
Oct 31 03:43:27.104485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104457    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca" pod "3e19927c-1a25-4bb0-87d4-5c92b8cbc614" (UID: "3e19927c-1a25-4bb0-87d4-5c92b8cbc614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:27.104947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.104910    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:27.106398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.106377    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp" pod "3e19927c-1a25-4bb0-87d4-5c92b8cbc614" (UID: "3e19927c-1a25-4bb0-87d4-5c92b8cbc614"). InnerVolumeSpecName "kube-api-access-pg2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:27.106520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.106464    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs" pod "3e19927c-1a25-4bb0-87d4-5c92b8cbc614" (UID: "3e19927c-1a25-4bb0-87d4-5c92b8cbc614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:27.106630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.106614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls" pod "3e19927c-1a25-4bb0-87d4-5c92b8cbc614" (UID: "3e19927c-1a25-4bb0-87d4-5c92b8cbc614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:27.206498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.206463    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:27.206498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.206494    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2tp\" (UniqueName: \"kubernetes.io/projected/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-kube-api-access-pg2tp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:27.206498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.206503    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e19927c-1a25-4bb0-87d4-5c92b8cbc614-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:27.478203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.478171    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:43:27.482058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.482028    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7a7f22-64b9cdd9-qjt6k"]
Oct 31 03:43:27.924438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:27.924382    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" podUID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" containerName="payload-processing" containerID="cri-o://a2cb0fd10b37b93adf4fa3cd7f18f57b5dc61bcd534aa2599c19787419efc6aa" gracePeriod=25
Oct 31 03:43:28.036409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.036355    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" podUID="2585a498-2c01-4528-88f6-3e64d3c25dce" containerName="payload-pre-processing" containerID="cri-o://4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0" gracePeriod=25
Oct 31 03:43:28.072112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.072075    5831 generic.go:356] "Generic (PLEG): container finished" podID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" containerID="a2cb0fd10b37b93adf4fa3cd7f18f57b5dc61bcd534aa2599c19787419efc6aa" exitCode=0
Oct 31 03:43:28.072291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.072143    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" event={"ID":"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b","Type":"ContainerDied","Data":"a2cb0fd10b37b93adf4fa3cd7f18f57b5dc61bcd534aa2599c19787419efc6aa"}
Oct 31 03:43:28.138933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.138892    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" path="/var/lib/kubelet/pods/3e19927c-1a25-4bb0-87d4-5c92b8cbc614/volumes"
Oct 31 03:43:28.222427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.222358    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:43:28.332351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.332325    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:43:28.341544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.341500    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\" (UniqueName: \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\") pod \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") "
Oct 31 03:43:28.341706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.341658    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\") pod \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\" (UID: \"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b\") "
Oct 31 03:43:28.342079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.342046    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume" pod "fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" (UID: "fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:28.342521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.342501    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:28.343946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.343925    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b" pod "fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" (UID: "fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b"). InnerVolumeSpecName "kube-api-access-5rx2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:28.442993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.442960    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\" (UniqueName: \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\") pod \"2585a498-2c01-4528-88f6-3e64d3c25dce\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") "
Oct 31 03:43:28.442993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.443004    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\") pod \"2585a498-2c01-4528-88f6-3e64d3c25dce\" (UID: \"2585a498-2c01-4528-88f6-3e64d3c25dce\") "
Oct 31 03:43:28.443397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.443373    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume" pod "2585a498-2c01-4528-88f6-3e64d3c25dce" (UID: "2585a498-2c01-4528-88f6-3e64d3c25dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:28.443598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.443580    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rx2b\" (UniqueName: \"kubernetes.io/projected/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b-kube-api-access-5rx2b\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:28.443664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.443604    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2585a498-2c01-4528-88f6-3e64d3c25dce-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:28.445260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.445238    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd" pod "2585a498-2c01-4528-88f6-3e64d3c25dce" (UID: "2585a498-2c01-4528-88f6-3e64d3c25dce"). InnerVolumeSpecName "kube-api-access-xk6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:28.554586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.552607    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6fd\" (UniqueName: \"kubernetes.io/projected/2585a498-2c01-4528-88f6-3e64d3c25dce-kube-api-access-xk6fd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:28.939523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.939489    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:43:28.940500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940475    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" containerName="maas-api"
Oct 31 03:43:28.940669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940502    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" containerName="maas-api"
Oct 31 03:43:28.940669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940524    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" containerName="payload-processing"
Oct 31 03:43:28.940669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940550    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" containerName="payload-processing"
Oct 31 03:43:28.940669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940572    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2585a498-2c01-4528-88f6-3e64d3c25dce" containerName="payload-pre-processing"
Oct 31 03:43:28.940669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940580    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2585a498-2c01-4528-88f6-3e64d3c25dce" containerName="payload-pre-processing"
Oct 31 03:43:28.940943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940773    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e19927c-1a25-4bb0-87d4-5c92b8cbc614" containerName="maas-api"
Oct 31 03:43:28.940943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940787    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" containerName="payload-processing"
Oct 31 03:43:28.940943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.940799    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2585a498-2c01-4528-88f6-3e64d3c25dce" containerName="payload-pre-processing"
Oct 31 03:43:28.945469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:28.945122    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.005079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.005050    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:43:29.023566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.018839    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dab9ee-dockercfg-h5bzs\""
Oct 31 03:43:29.045361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.045331    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dab9ee\""
Oct 31 03:43:29.069956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.069925    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4nm\" (UniqueName: \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.070116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.070087    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.082932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.082894    5831 generic.go:356] "Generic (PLEG): container finished" podID="2585a498-2c01-4528-88f6-3e64d3c25dce" containerID="4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0" exitCode=0
Oct 31 03:43:29.083096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.083003    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"
Oct 31 03:43:29.083096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.083050    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" event={"ID":"2585a498-2c01-4528-88f6-3e64d3c25dce","Type":"ContainerDied","Data":"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"}
Oct 31 03:43:29.083273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.083105    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq" event={"ID":"2585a498-2c01-4528-88f6-3e64d3c25dce","Type":"ContainerDied","Data":"e292daeb025a2bb1756987af6a1df70535c4fac77fd159e72523d0e7d4390fe3"}
Oct 31 03:43:29.083273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.083131    5831 scope.go:122] "RemoveContainer" containerID="4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"
Oct 31 03:43:29.086593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.086567    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc" event={"ID":"fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b","Type":"ContainerDied","Data":"0fb6f3d60bddc1bab3fe6dbd716340177c016f60829d17ca477fbd807bc78c3e"}
Oct 31 03:43:29.086593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.086586    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"
Oct 31 03:43:29.106231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.106206    5831 scope.go:122] "RemoveContainer" containerID="4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"
Oct 31 03:43:29.106728 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:29.106705    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0\": container with ID starting with 4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0 not found: ID does not exist" containerID="4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"
Oct 31 03:43:29.106843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.106738    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0"} err="failed to get container status \"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0\": rpc error: code = NotFound desc = could not find container \"4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0\": container with ID starting with 4b29a4342e52b90a5fb11c4bc4739468fd64d6312f7a26e528ec6332f0f1ceb0 not found: ID does not exist"
Oct 31 03:43:29.106843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.106756    5831 scope.go:122] "RemoveContainer" containerID="a2cb0fd10b37b93adf4fa3cd7f18f57b5dc61bcd534aa2599c19787419efc6aa"
Oct 31 03:43:29.185559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.177317    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4nm\" (UniqueName: \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.185559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.177490    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.185559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.178554    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.270103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.267457    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4nm\" (UniqueName: \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\") pod \"payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.270103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.269233    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:43:29.277554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.274518    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7a7f22-cd757844c-4txtc"]
Oct 31 03:43:29.377844 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.377778    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:43:29.382683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.382647    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7a7f22-78dfd4645f-9hfcq"]
Oct 31 03:43:29.562227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.562141    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:29.943487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:29.943463    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:43:29.945333 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:29.945303    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f67d38_d3f3_4c15_8c6f_3ff689fe9625.slice/crio-9a9737ebfadaaccf3f7f6553c20287ef8590fb938780791aae3a9b3eb730c143 WatchSource:0}: Error finding container 9a9737ebfadaaccf3f7f6553c20287ef8590fb938780791aae3a9b3eb730c143: Status 404 returned error can't find the container with id 9a9737ebfadaaccf3f7f6553c20287ef8590fb938780791aae3a9b3eb730c143
Oct 31 03:43:30.100876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.100831    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" event={"ID":"19f67d38-d3f3-4c15-8c6f-3ff689fe9625","Type":"ContainerStarted","Data":"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"}
Oct 31 03:43:30.101083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.100932    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" event={"ID":"19f67d38-d3f3-4c15-8c6f-3ff689fe9625","Type":"ContainerStarted","Data":"9a9737ebfadaaccf3f7f6553c20287ef8590fb938780791aae3a9b3eb730c143"}
Oct 31 03:43:30.101161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.101088    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:30.136594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.136517    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2585a498-2c01-4528-88f6-3e64d3c25dce" path="/var/lib/kubelet/pods/2585a498-2c01-4528-88f6-3e64d3c25dce/volumes"
Oct 31 03:43:30.137008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.136994    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b" path="/var/lib/kubelet/pods/fbd0bc53-c7c9-4672-9ac7-549b3ab8d16b/volumes"
Oct 31 03:43:30.225721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:30.225615    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" podStartSLOduration=2.22559879 podStartE2EDuration="2.22559879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:30.218476921 +0000 UTC m=+2160.233038641" watchObservedRunningTime="2026-10-31 03:43:30.22559879 +0000 UTC m=+2160.240160497"
Oct 31 03:43:31.681100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:31.681056    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:43:31.681627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:31.681312    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" containerID="cri-o://e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb" gracePeriod=30
Oct 31 03:43:32.566126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:32.566088    5831 patch_prober.go:29] interesting pod/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:43:32.566323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:32.566152    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:43:33.401560 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.397596    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:43:33.494207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.494170    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:43:33.494497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.494310    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.544877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.544841    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-14600f-openshift-default-dockercfg-k8hjr\""
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.627786    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7h7\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.627872    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.627947    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628000    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628106    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628153    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628199    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628224    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.630869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.628255    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729336    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7h7\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729410    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729467    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729511    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729644    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729696    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729754    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729781    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.729819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.729812    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.730072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.730001    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.730171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.730136    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.730246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.730211    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.730325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.730303    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.730635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.730610    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.732331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.732312    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.732512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.732491    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.789248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.789204    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.789427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.789399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7h7\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\") pod \"e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") " pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:33.816345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:33.816317    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:34.235960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.235924    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:43:34.241678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.241612    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.321553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.312084    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:43:34.321553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.312396    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d22078\""
Oct 31 03:43:34.321553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.313249    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d22078-dockercfg-hfrkt\""
Oct 31 03:43:34.351555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.339338    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmgs\" (UniqueName: \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.351555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.339586    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.424196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.419200    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:34.424196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.419275    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:34.424196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.419308    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:43:34.424196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.421119    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:43:34.441944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.441876    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.442121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.441962    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmgs\" (UniqueName: \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.443123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.443090    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.505396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.505365    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmgs\" (UniqueName: \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\") pod \"payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.557191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.557152    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:34.948662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:34.948631    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:43:34.950727 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:34.950697    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b68dc18_e915_4c58_9693_42796ac20fa5.slice/crio-f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe WatchSource:0}: Error finding container f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe: Status 404 returned error can't find the container with id f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe
Oct 31 03:43:35.146044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.145947    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerStarted","Data":"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"}
Oct 31 03:43:35.146044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.146000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerStarted","Data":"e4d0bab86036ec3e9219a92f68d573a14b552dde68855b7c2639d4db5de35c1f"}
Oct 31 03:43:35.147568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.147518    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" event={"ID":"8b68dc18-e915-4c58-9693-42796ac20fa5","Type":"ContainerStarted","Data":"3b9efe4c8071ab35dcf8553281638ff9a48cb0fd329563a0b07f13a30811a049"}
Oct 31 03:43:35.147683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.147587    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" event={"ID":"8b68dc18-e915-4c58-9693-42796ac20fa5","Type":"ContainerStarted","Data":"f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe"}
Oct 31 03:43:35.147683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.147664    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:35.260044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.259989    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podStartSLOduration=2.259976009 podStartE2EDuration="2.259976009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:35.256622691 +0000 UTC m=+2165.271184401" watchObservedRunningTime="2026-10-31 03:43:35.259976009 +0000 UTC m=+2165.274537717"
Oct 31 03:43:35.363699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.363635    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" podStartSLOduration=1.363620317 podStartE2EDuration="1.363620317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:35.361819196 +0000 UTC m=+2165.376380895" watchObservedRunningTime="2026-10-31 03:43:35.363620317 +0000 UTC m=+2165.378182025"
Oct 31 03:43:35.816882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.816848    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:35.823661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:35.823634    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:36.155671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:36.155589    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:36.156440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:36.156421    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:43:36.977523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:36.977496    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:43:37.081159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081120    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081206    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081293    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081334    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081402    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081471    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081511    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081636    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081642    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:37.081778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081726    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\") pod \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\" (UID: \"5a4f7fd6-8d21-44f2-be89-54ca99477da6\") "
Oct 31 03:43:37.081778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.081764    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:37.082039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082012    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:37.082317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082289    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:37.082494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082471    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:37.082775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082505    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.082775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082570    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.082775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082586    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.082775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.082600    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.083781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.083758    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:37.084570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.084507    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "kube-api-access-nv6kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:37.084675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.084598    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:37.084749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.084636    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token" pod "5a4f7fd6-8d21-44f2-be89-54ca99477da6" (UID: "5a4f7fd6-8d21-44f2-be89-54ca99477da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:37.165446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.165411    5831 generic.go:356] "Generic (PLEG): container finished" podID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerID="e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb" exitCode=0
Oct 31 03:43:37.165674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.165567    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"
Oct 31 03:43:37.165674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.165581    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerDied","Data":"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"}
Oct 31 03:43:37.165674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.165644    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp" event={"ID":"5a4f7fd6-8d21-44f2-be89-54ca99477da6","Type":"ContainerDied","Data":"1efc2a12f96956618e772451adb92439ff074c33c325f3321d73f6235eea1c58"}
Oct 31 03:43:37.165674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.165674    5831 scope.go:122] "RemoveContainer" containerID="e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"
Oct 31 03:43:37.178927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.178907    5831 scope.go:122] "RemoveContainer" containerID="706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"
Oct 31 03:43:37.184115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.184088    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.184115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.184116    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.184266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.184130    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6kp\" (UniqueName: \"kubernetes.io/projected/5a4f7fd6-8d21-44f2-be89-54ca99477da6-kube-api-access-nv6kp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.184266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.184144    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4f7fd6-8d21-44f2-be89-54ca99477da6-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.184266 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.184157    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4f7fd6-8d21-44f2-be89-54ca99477da6-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:37.191047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.190943    5831 scope.go:122] "RemoveContainer" containerID="e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"
Oct 31 03:43:37.191317 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:37.191298    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb\": container with ID starting with e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb not found: ID does not exist" containerID="e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"
Oct 31 03:43:37.191400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.191330    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb"} err="failed to get container status \"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb\": rpc error: code = NotFound desc = could not find container \"e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb\": container with ID starting with e277794bc89d6df5d2eb57e326e19d3d1771f5aa4c0fa3917a011e4b1cd790eb not found: ID does not exist"
Oct 31 03:43:37.191400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.191356    5831 scope.go:122] "RemoveContainer" containerID="706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"
Oct 31 03:43:37.191764 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:37.191745    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff\": container with ID starting with 706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff not found: ID does not exist" containerID="706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"
Oct 31 03:43:37.191857 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.191774    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff"} err="failed to get container status \"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff\": rpc error: code = NotFound desc = could not find container \"706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff\": container with ID starting with 706b25e32556a4c7e737043eeafc762b8c75db1964edb2d0b06a6a50866941ff not found: ID does not exist"
Oct 31 03:43:37.322453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.322350    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:43:37.327741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:37.327710    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7a7f22-openshift-default-5d767fffb9-szvkp"]
Oct 31 03:43:38.135945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:38.135910    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" path="/var/lib/kubelet/pods/5a4f7fd6-8d21-44f2-be89-54ca99477da6/volumes"
Oct 31 03:43:41.114350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:41.114321    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:43:42.115557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.112847    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118358    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118387    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118420    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118429    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118650    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.120235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.118662    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4f7fd6-8d21-44f2-be89-54ca99477da6" containerName="istio-proxy"
Oct 31 03:43:42.150975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.150945    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.196586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.192971    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:43:42.202503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.202473    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-14600f\""
Oct 31 03:43:42.209630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.202863    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-14600f-dockercfg-s8j8h\""
Oct 31 03:43:42.254077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.254048    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqvw\" (UniqueName: \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.254268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.254175    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.293710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.293661    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:43:42.302281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.300300    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.330627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.330597    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:43:42.355149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.355119    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.355360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.355340    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqvw\" (UniqueName: \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.355924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.355902    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.411267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.411190    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqvw\" (UniqueName: \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\") pod \"payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.456691 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.456651    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.456843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.456731    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgr6w\" (UniqueName: \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.479576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.479541    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:42.561558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.558873    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.561558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.558972    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgr6w\" (UniqueName: \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.561558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.560191    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.612358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.612318    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgr6w\" (UniqueName: \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\") pod \"payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.617288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.617259    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:42.872430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:42.872395    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:43:42.873869 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:42.873833    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1536fbf_903f_47e8_8b35_fd283c22a34b.slice/crio-c5d51d4224190862b3afd2834057fda7250ae646af81418c81c1817f9b421d62 WatchSource:0}: Error finding container c5d51d4224190862b3afd2834057fda7250ae646af81418c81c1817f9b421d62: Status 404 returned error can't find the container with id c5d51d4224190862b3afd2834057fda7250ae646af81418c81c1817f9b421d62
Oct 31 03:43:42.982873 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:43:42.982833    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:43:43.001115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.001091    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:43:43.003455 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:43:43.003429    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fe3b91_96e1_4dcb_b1f4_9abdb0f61077.slice/crio-30598c53a64c920ea194075d03a814881665c944d759c095331da9a20ca7ff43 WatchSource:0}: Error finding container 30598c53a64c920ea194075d03a814881665c944d759c095331da9a20ca7ff43: Status 404 returned error can't find the container with id 30598c53a64c920ea194075d03a814881665c944d759c095331da9a20ca7ff43
Oct 31 03:43:43.236683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.236580    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" event={"ID":"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077","Type":"ContainerStarted","Data":"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"}
Oct 31 03:43:43.237148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.236683    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:43:43.237148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.236711    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" event={"ID":"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077","Type":"ContainerStarted","Data":"30598c53a64c920ea194075d03a814881665c944d759c095331da9a20ca7ff43"}
Oct 31 03:43:43.238424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.238397    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" event={"ID":"a1536fbf-903f-47e8-8b35-fd283c22a34b","Type":"ContainerStarted","Data":"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"}
Oct 31 03:43:43.238549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.238446    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" event={"ID":"a1536fbf-903f-47e8-8b35-fd283c22a34b","Type":"ContainerStarted","Data":"c5d51d4224190862b3afd2834057fda7250ae646af81418c81c1817f9b421d62"}
Oct 31 03:43:43.238549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.238483    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:43.343874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.343822    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" podStartSLOduration=1.343807161 podStartE2EDuration="1.343807161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:43.342680963 +0000 UTC m=+2173.357242670" watchObservedRunningTime="2026-10-31 03:43:43.343807161 +0000 UTC m=+2173.358368869"
Oct 31 03:43:43.452224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:43.452168    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" podStartSLOduration=1.452150903 podStartE2EDuration="1.452150903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:43.449486802 +0000 UTC m=+2173.464048510" watchObservedRunningTime="2026-10-31 03:43:43.452150903 +0000 UTC m=+2173.466712611"
Oct 31 03:43:46.158307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:46.158272    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:43:51.510419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:51.510383    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:43:54.248240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:54.248206    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:43:54.248717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:43:54.248280    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:44:03.086895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.086856    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:44:03.093251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.093225    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.144048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.144014    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:44:03.186207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.186174    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c561e3\""
Oct 31 03:44:03.186422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.186273    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c561e3-dockercfg-qwcq6\""
Oct 31 03:44:03.265429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.265387    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26sv\" (UniqueName: \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.265662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.265647    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.367349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.367252    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.367587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.367380    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26sv\" (UniqueName: \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.368025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.368005    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.428699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.428664    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26sv\" (UniqueName: \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\") pod \"payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:03.711841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:03.711799    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:04.118966 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:04.118932    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:44:04.123159 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:44:04.123112    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda325465a_6593_4466_b0a4_e260b56e298e.slice/crio-8ba6f0a1029a337c2e6978682b2cb619aefd5d72aa581ff3a346ed7c700f4d89 WatchSource:0}: Error finding container 8ba6f0a1029a337c2e6978682b2cb619aefd5d72aa581ff3a346ed7c700f4d89: Status 404 returned error can't find the container with id 8ba6f0a1029a337c2e6978682b2cb619aefd5d72aa581ff3a346ed7c700f4d89
Oct 31 03:44:04.411887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:04.411787    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" event={"ID":"a325465a-6593-4466-b0a4-e260b56e298e","Type":"ContainerStarted","Data":"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"}
Oct 31 03:44:04.411887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:04.411840    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" event={"ID":"a325465a-6593-4466-b0a4-e260b56e298e","Type":"ContainerStarted","Data":"8ba6f0a1029a337c2e6978682b2cb619aefd5d72aa581ff3a346ed7c700f4d89"}
Oct 31 03:44:04.412144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:04.411939    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:04.520024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:04.519964    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" podStartSLOduration=1.519948836 podStartE2EDuration="1.519948836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:04.51663139 +0000 UTC m=+2194.531193100" watchObservedRunningTime="2026-10-31 03:44:04.519948836 +0000 UTC m=+2194.534510544"
Oct 31 03:44:12.980974 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:44:12.980936    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:44:15.424336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:15.424304    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:44:23.158546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:23.158500    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:44:27.975299 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:27.975263    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:44:27.980433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:27.980409    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.075002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.074965    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-dab9ee-serving-cert\""
Oct 31 03:44:28.078439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.078417    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:44:28.102240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.102202    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.102363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.102295    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdb6\" (UniqueName: \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.102406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.102370    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.102503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.102489    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.203986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.203948    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.204202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.204012    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdb6\" (UniqueName: \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.204202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.204053    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.204202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.204150    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.204399 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:44:28.204316    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-serving-cert: secret "maas-api-e2e-praxis-a-dab9ee-serving-cert" not found
Oct 31 03:44:28.204458 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:44:28.204411    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls podName:8cf79f28-966b-4b87-8261-dc71bd838c47 nodeName:}" failed. No retries permitted until 2026-10-31 03:44:28.704380924 +0000 UTC m=+2218.718942632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls") pod "maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" (UID: "8cf79f28-966b-4b87-8261-dc71bd838c47") : secret "maas-api-e2e-praxis-a-dab9ee-serving-cert" not found
Oct 31 03:44:28.204873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.204854    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.206792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.206768    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.277796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.277647    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdb6\" (UniqueName: \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.711888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.711848    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.714671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.714645    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:28.895600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:28.895562    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:29.273956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:29.273877    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:44:29.276215 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:44:29.276189    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf79f28_966b_4b87_8261_dc71bd838c47.slice/crio-b06eb6a74309e7bbda1e34a0c64f21ca55592dd41763a7b2957cb1fd042f2f90 WatchSource:0}: Error finding container b06eb6a74309e7bbda1e34a0c64f21ca55592dd41763a7b2957cb1fd042f2f90: Status 404 returned error can't find the container with id b06eb6a74309e7bbda1e34a0c64f21ca55592dd41763a7b2957cb1fd042f2f90
Oct 31 03:44:29.607877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:29.607783    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" event={"ID":"8cf79f28-966b-4b87-8261-dc71bd838c47","Type":"ContainerStarted","Data":"b06eb6a74309e7bbda1e34a0c64f21ca55592dd41763a7b2957cb1fd042f2f90"}
Oct 31 03:44:30.618581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:30.618430    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" event={"ID":"8cf79f28-966b-4b87-8261-dc71bd838c47","Type":"ContainerStarted","Data":"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"}
Oct 31 03:44:30.618581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:30.618562    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:30.728626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:30.728563    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" podStartSLOduration=2.745172949 podStartE2EDuration="3.728512246s" totalImagesPullingTime="983.339297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:30.725230733 +0000 UTC m=+2220.739792462" watchObservedRunningTime="2026-10-31 03:44:30.728512246 +0000 UTC m=+2220.743073979"
Oct 31 03:44:36.634492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:36.634455    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:44:42.984064 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:44:42.984010    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:44:52.806077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.806027    5831 generic.go:356] "Generic (PLEG): container finished" podID="08bf809b-e77c-4145-8949-b4935befb5e7" containerID="a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2" exitCode=0
Oct 31 03:44:52.806554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.806114    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerDied","Data":"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"}
Oct 31 03:44:52.806619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.806595    5831 scope.go:122] "RemoveContainer" containerID="a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"
Oct 31 03:44:52.808191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.808164    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:44:52.808267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.808236    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:44:52.808308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:52.808269    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:44:53.030721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:53.030682    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:44:53.822940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:53.822898    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerStarted","Data":"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"}
Oct 31 03:44:54.613923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:54.613883    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:44:54.615359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:54.615326    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 31 03:44:54.615496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:54.615396    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 31 03:44:55.614070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:55.614033    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 31 03:44:55.614514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:55.614102    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 31 03:44:56.613505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:56.613472    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 31 03:44:56.613726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:56.613556    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 31 03:44:57.614108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:57.614071    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 31 03:44:57.614571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:57.614122    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 31 03:44:58.614202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:58.614106    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 31 03:44:58.614202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:58.614184    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 31 03:44:59.618394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:59.618362    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:44:59.618925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:59.618865    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:44:59.619801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:44:59.619776    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:45:00.162097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.162064    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:00.167171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.167144    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:00.167313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.167261    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.171242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.171223    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.171615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.171596    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:00.175813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.175795    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.190826 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.190804    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:00.194989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.194966    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.195668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.195643    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:00.212883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.212857    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.217586 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.217561    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:00.221011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.220959    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:45:00.221989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.221971    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.257131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.257101    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:00.266620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.266590    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:00.268304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.268275    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:00.306872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.306832    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:00.312759 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.312728    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:00.315548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315504    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpm9\" (UniqueName: \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.315696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315568    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.315696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315608    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.315696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315644    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbgm\" (UniqueName: \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.315884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315799    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gh6\" (UniqueName: \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.315946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315895    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.315946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315919    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjmr\" (UniqueName: \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.316071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315946    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kghd\" (UniqueName: \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.316071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.315993    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.316071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.316049    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvd9c\" (UniqueName: \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.316241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.316070    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.316241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.316198    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.319772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.319748    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:00.417406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417313    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.417406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417350    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjmr\" (UniqueName: \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.417406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kghd\" (UniqueName: \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.417707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417508    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.417707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417632    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvd9c\" (UniqueName: \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.417707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417666    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.417888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417755    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.417888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417830    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpm9\" (UniqueName: \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.417888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.417859    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.418230 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418200    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.418355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418266    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbgm\" (UniqueName: \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.418355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418304    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.418478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418352    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gh6\" (UniqueName: \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.418559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418479    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.418707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418687    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.418784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418745    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.418958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.418939    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.419070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.419054    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.472686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.472645    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvd9c\" (UniqueName: \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.472686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.472657    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kghd\" (UniqueName: \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.472686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.472678    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbgm\" (UniqueName: \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\") pod \"maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.472686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.472645    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpm9\" (UniqueName: \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\") pod \"maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.472977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.472660    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gh6\" (UniqueName: \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\") pod \"maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.473094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.473077    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjmr\" (UniqueName: \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.483682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.483661    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:00.492285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.492264    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:00.513685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.513660    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:00.520869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.516709    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:00.528103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.527808    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:00.538505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.538221    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:00.871073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.871041    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:00.872635 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:00.872602    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fed8bef_3530_4d58_97ed_f4d7d65da249.slice/crio-c3f7584b526fa31b2f4c76456d73d5a7c8024ec1b72b07c79ce5d1d3d2d44f27 WatchSource:0}: Error finding container c3f7584b526fa31b2f4c76456d73d5a7c8024ec1b72b07c79ce5d1d3d2d44f27: Status 404 returned error can't find the container with id c3f7584b526fa31b2f4c76456d73d5a7c8024ec1b72b07c79ce5d1d3d2d44f27
Oct 31 03:45:00.913120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.913094    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:00.915944 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:00.915918    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdb859a_40c2_492d_a0dc_d5c55bdaff69.slice/crio-a6a83382d856d1978acd9fa1c268e6394415fad6254f08dd4897af98545259ed WatchSource:0}: Error finding container a6a83382d856d1978acd9fa1c268e6394415fad6254f08dd4897af98545259ed: Status 404 returned error can't find the container with id a6a83382d856d1978acd9fa1c268e6394415fad6254f08dd4897af98545259ed
Oct 31 03:45:00.927480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.927433    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:00.928808 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:00.928784    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb10b68_777f_4856_9685_000126754651.slice/crio-6329cce639e8cef920282a64f19d2aab6eac61470d2a74493438305f0fdafe35 WatchSource:0}: Error finding container 6329cce639e8cef920282a64f19d2aab6eac61470d2a74493438305f0fdafe35: Status 404 returned error can't find the container with id 6329cce639e8cef920282a64f19d2aab6eac61470d2a74493438305f0fdafe35
Oct 31 03:45:00.975773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.975743    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:00.977159 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:00.977134    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a77297_dcac_477b_95e7_489a4bd10cd1.slice/crio-0626a640bca0984c7c619c63a0bdf8dba07f8638f44422ec47d8dd241339fe82 WatchSource:0}: Error finding container 0626a640bca0984c7c619c63a0bdf8dba07f8638f44422ec47d8dd241339fe82: Status 404 returned error can't find the container with id 0626a640bca0984c7c619c63a0bdf8dba07f8638f44422ec47d8dd241339fe82
Oct 31 03:45:00.996905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:00.996875    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:01.021101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.021077    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:01.084829 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:01.084795    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83738f85_542f_4f9b_a75c_68f6d518ebe2.slice/crio-546b4d2401cafb4c72fe0be11673831d3122b761ccba0fb0205911f327ec4899 WatchSource:0}: Error finding container 546b4d2401cafb4c72fe0be11673831d3122b761ccba0fb0205911f327ec4899: Status 404 returned error can't find the container with id 546b4d2401cafb4c72fe0be11673831d3122b761ccba0fb0205911f327ec4899
Oct 31 03:45:01.085894 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:01.085867    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae15a21_812e_420f_9a4e_2acdd757cb51.slice/crio-004fa55dfb4ecebd5c45c97c5037d28d353daf5db4c87c8320f68d01ec85c93c WatchSource:0}: Error finding container 004fa55dfb4ecebd5c45c97c5037d28d353daf5db4c87c8320f68d01ec85c93c: Status 404 returned error can't find the container with id 004fa55dfb4ecebd5c45c97c5037d28d353daf5db4c87c8320f68d01ec85c93c
Oct 31 03:45:01.900384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.900354    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerStarted","Data":"e3b227b4533a86ad38d42327166284bde37207e97cfc6f7b51c798a16e89f191"}
Oct 31 03:45:01.900874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.900419    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerStarted","Data":"0626a640bca0984c7c619c63a0bdf8dba07f8638f44422ec47d8dd241339fe82"}
Oct 31 03:45:01.902248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.902217    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerStarted","Data":"bb2597933f876e4abadff74cced686eb8de82b7794ded49e405a7d386c355154"}
Oct 31 03:45:01.902389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.902264    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerStarted","Data":"546b4d2401cafb4c72fe0be11673831d3122b761ccba0fb0205911f327ec4899"}
Oct 31 03:45:01.903967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.903944    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerStarted","Data":"e536b50beb370e8e42d65ff1de83437229df57e56c66e874129518b19d8baa58"}
Oct 31 03:45:01.904097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.903988    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerStarted","Data":"004fa55dfb4ecebd5c45c97c5037d28d353daf5db4c87c8320f68d01ec85c93c"}
Oct 31 03:45:01.905933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.905905    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerStarted","Data":"d3341c9d8b8c902a3136f74df2a506b4e2eb3574f49e9b0ae65796d8c04e2036"}
Oct 31 03:45:01.906037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.905950    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerStarted","Data":"c3f7584b526fa31b2f4c76456d73d5a7c8024ec1b72b07c79ce5d1d3d2d44f27"}
Oct 31 03:45:01.907588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.907565    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerStarted","Data":"8d46d28686390c29867306a0de296bc76dc1f7089cf9c2536c71811c87a2ce27"}
Oct 31 03:45:01.907687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.907613    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerStarted","Data":"a6a83382d856d1978acd9fa1c268e6394415fad6254f08dd4897af98545259ed"}
Oct 31 03:45:01.909328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.909308    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerStarted","Data":"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"}
Oct 31 03:45:01.909437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.909342    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerStarted","Data":"6329cce639e8cef920282a64f19d2aab6eac61470d2a74493438305f0fdafe35"}
Oct 31 03:45:01.909516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:01.909497    5831 scope.go:122] "RemoveContainer" containerID="6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"
Oct 31 03:45:02.007394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.007338    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" podStartSLOduration=2.007325274 podStartE2EDuration="2.007325274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.003627984 +0000 UTC m=+2252.018189692" watchObservedRunningTime="2026-10-31 03:45:02.007325274 +0000 UTC m=+2252.021886981"
Oct 31 03:45:02.114259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.114209    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" podStartSLOduration=2.114194887 podStartE2EDuration="2.114194887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.111161312 +0000 UTC m=+2252.125723019" watchObservedRunningTime="2026-10-31 03:45:02.114194887 +0000 UTC m=+2252.128756595"
Oct 31 03:45:02.218362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.218311    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" podStartSLOduration=2.218297421 podStartE2EDuration="2.218297421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.216283604 +0000 UTC m=+2252.230845312" watchObservedRunningTime="2026-10-31 03:45:02.218297421 +0000 UTC m=+2252.232859129"
Oct 31 03:45:02.323985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.323942    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" podStartSLOduration=2.323927708 podStartE2EDuration="2.323927708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.320626083 +0000 UTC m=+2252.335187792" watchObservedRunningTime="2026-10-31 03:45:02.323927708 +0000 UTC m=+2252.338489415"
Oct 31 03:45:02.529979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.529917    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" podStartSLOduration=2.529897543 podStartE2EDuration="2.529897543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.528518318 +0000 UTC m=+2252.543080018" watchObservedRunningTime="2026-10-31 03:45:02.529897543 +0000 UTC m=+2252.544459255"
Oct 31 03:45:02.922706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.922669    5831 generic.go:356] "Generic (PLEG): container finished" podID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerID="8d46d28686390c29867306a0de296bc76dc1f7089cf9c2536c71811c87a2ce27" exitCode=60
Oct 31 03:45:02.923219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.922761    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerDied","Data":"8d46d28686390c29867306a0de296bc76dc1f7089cf9c2536c71811c87a2ce27"}
Oct 31 03:45:02.923219 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.923067    5831 scope.go:122] "RemoveContainer" containerID="8d46d28686390c29867306a0de296bc76dc1f7089cf9c2536c71811c87a2ce27"
Oct 31 03:45:02.925512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925341    5831 generic.go:356] "Generic (PLEG): container finished" podID="cfb10b68-777f-4856-9685-000126754651" containerID="6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61" exitCode=60
Oct 31 03:45:02.925512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925365    5831 generic.go:356] "Generic (PLEG): container finished" podID="cfb10b68-777f-4856-9685-000126754651" containerID="36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2" exitCode=60
Oct 31 03:45:02.925512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925424    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerDied","Data":"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"}
Oct 31 03:45:02.925512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925496    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerDied","Data":"36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2"}
Oct 31 03:45:02.925808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925574    5831 scope.go:122] "RemoveContainer" containerID="6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"
Oct 31 03:45:02.925869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.925820    5831 scope.go:122] "RemoveContainer" containerID="36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2"
Oct 31 03:45:02.926145 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:02.926122    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8_odh-ai-gateway-infra(cfb10b68-777f-4856-9685-000126754651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" podUID="cfb10b68-777f-4856-9685-000126754651"
Oct 31 03:45:02.928664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.928403    5831 generic.go:356] "Generic (PLEG): container finished" podID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerID="e3b227b4533a86ad38d42327166284bde37207e97cfc6f7b51c798a16e89f191" exitCode=60
Oct 31 03:45:02.928664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.928487    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerDied","Data":"e3b227b4533a86ad38d42327166284bde37207e97cfc6f7b51c798a16e89f191"}
Oct 31 03:45:02.928791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.928776    5831 scope.go:122] "RemoveContainer" containerID="e3b227b4533a86ad38d42327166284bde37207e97cfc6f7b51c798a16e89f191"
Oct 31 03:45:02.930410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.930391    5831 generic.go:356] "Generic (PLEG): container finished" podID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerID="bb2597933f876e4abadff74cced686eb8de82b7794ded49e405a7d386c355154" exitCode=60
Oct 31 03:45:02.930519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.930479    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerDied","Data":"bb2597933f876e4abadff74cced686eb8de82b7794ded49e405a7d386c355154"}
Oct 31 03:45:02.930749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.930733    5831 scope.go:122] "RemoveContainer" containerID="bb2597933f876e4abadff74cced686eb8de82b7794ded49e405a7d386c355154"
Oct 31 03:45:02.932236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.932210    5831 generic.go:356] "Generic (PLEG): container finished" podID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerID="e536b50beb370e8e42d65ff1de83437229df57e56c66e874129518b19d8baa58" exitCode=60
Oct 31 03:45:02.932331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.932247    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerDied","Data":"e536b50beb370e8e42d65ff1de83437229df57e56c66e874129518b19d8baa58"}
Oct 31 03:45:02.932519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.932500    5831 scope.go:122] "RemoveContainer" containerID="e536b50beb370e8e42d65ff1de83437229df57e56c66e874129518b19d8baa58"
Oct 31 03:45:02.933979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.933958    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerID="d3341c9d8b8c902a3136f74df2a506b4e2eb3574f49e9b0ae65796d8c04e2036" exitCode=60
Oct 31 03:45:02.934075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.934026    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerDied","Data":"d3341c9d8b8c902a3136f74df2a506b4e2eb3574f49e9b0ae65796d8c04e2036"}
Oct 31 03:45:02.934285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.934270    5831 scope.go:122] "RemoveContainer" containerID="d3341c9d8b8c902a3136f74df2a506b4e2eb3574f49e9b0ae65796d8c04e2036"
Oct 31 03:45:02.941961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.941936    5831 scope.go:122] "RemoveContainer" containerID="6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"
Oct 31 03:45:02.942255 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:02.942232    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61\": container with ID starting with 6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61 not found: ID does not exist" containerID="6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"
Oct 31 03:45:02.942349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:02.942259    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61"} err="failed to get container status \"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61\": rpc error: code = NotFound desc = could not find container \"6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61\": container with ID starting with 6adf79bcd81aa0fba6fc9eccde0d97d1ef0fff1bf6a7cb62d33a047a3c3c9c61 not found: ID does not exist"
Oct 31 03:45:03.944775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.944739    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerStarted","Data":"b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"}
Oct 31 03:45:03.946832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.946810    5831 scope.go:122] "RemoveContainer" containerID="36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2"
Oct 31 03:45:03.947122 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:03.947099    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8_odh-ai-gateway-infra(cfb10b68-777f-4856-9685-000126754651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" podUID="cfb10b68-777f-4856-9685-000126754651"
Oct 31 03:45:03.948582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.948558    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerStarted","Data":"7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"}
Oct 31 03:45:03.950445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.950420    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerStarted","Data":"b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"}
Oct 31 03:45:03.952199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.952179    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerStarted","Data":"632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"}
Oct 31 03:45:03.954043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:03.954013    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerStarted","Data":"ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"}
Oct 31 03:45:04.964649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.964615    5831 generic.go:356] "Generic (PLEG): container finished" podID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerID="632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d" exitCode=60
Oct 31 03:45:04.965142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.964698    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerDied","Data":"632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"}
Oct 31 03:45:04.965142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.964766    5831 scope.go:122] "RemoveContainer" containerID="e536b50beb370e8e42d65ff1de83437229df57e56c66e874129518b19d8baa58"
Oct 31 03:45:04.965142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.965010    5831 scope.go:122] "RemoveContainer" containerID="632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"
Oct 31 03:45:04.965362 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:04.965335    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6_odh-ai-gateway-infra(2ae15a21-812e-420f-9a4e-2acdd757cb51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51"
Oct 31 03:45:04.966883 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.966859    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerID="ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84" exitCode=60
Oct 31 03:45:04.966967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.966912    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerDied","Data":"ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"}
Oct 31 03:45:04.967200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.967186    5831 scope.go:122] "RemoveContainer" containerID="ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"
Oct 31 03:45:04.967491 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:04.967474    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j_odh-ai-gateway-infra(0fed8bef-3530-4d58-97ed-f4d7d65da249)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249"
Oct 31 03:45:04.968862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.968843    5831 generic.go:356] "Generic (PLEG): container finished" podID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerID="b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8" exitCode=60
Oct 31 03:45:04.968992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.968868    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerDied","Data":"b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"}
Oct 31 03:45:04.969179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.969164    5831 scope.go:122] "RemoveContainer" containerID="b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"
Oct 31 03:45:04.969460 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:04.969440    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv_odh-ai-gateway-infra(6fdb859a-40c2-492d-a0dc-d5c55bdaff69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69"
Oct 31 03:45:04.971071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.971048    5831 generic.go:356] "Generic (PLEG): container finished" podID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerID="7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72" exitCode=60
Oct 31 03:45:04.971159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.971140    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerDied","Data":"7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"}
Oct 31 03:45:04.971330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.971312    5831 scope.go:122] "RemoveContainer" containerID="7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"
Oct 31 03:45:04.971609 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:04.971588    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd_odh-ai-gateway-infra(05a77297-dcac-477b-95e7-489a4bd10cd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1"
Oct 31 03:45:04.973228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.973206    5831 generic.go:356] "Generic (PLEG): container finished" podID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerID="b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6" exitCode=60
Oct 31 03:45:04.973331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.973254    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerDied","Data":"b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"}
Oct 31 03:45:04.973572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.973551    5831 scope.go:122] "RemoveContainer" containerID="b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"
Oct 31 03:45:04.973876 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:04.973857    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn_odh-ai-gateway-infra(83738f85-542f-4f9b-a75c-68f6d518ebe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2"
Oct 31 03:45:04.980089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:04.980072    5831 scope.go:122] "RemoveContainer" containerID="d3341c9d8b8c902a3136f74df2a506b4e2eb3574f49e9b0ae65796d8c04e2036"
Oct 31 03:45:05.002552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.002512    5831 scope.go:122] "RemoveContainer" containerID="8d46d28686390c29867306a0de296bc76dc1f7089cf9c2536c71811c87a2ce27"
Oct 31 03:45:05.021436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.021407    5831 scope.go:122] "RemoveContainer" containerID="e3b227b4533a86ad38d42327166284bde37207e97cfc6f7b51c798a16e89f191"
Oct 31 03:45:05.035767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.035748    5831 scope.go:122] "RemoveContainer" containerID="bb2597933f876e4abadff74cced686eb8de82b7794ded49e405a7d386c355154"
Oct 31 03:45:05.983985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.983954    5831 scope.go:122] "RemoveContainer" containerID="7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"
Oct 31 03:45:05.984480 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:05.984325    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd_odh-ai-gateway-infra(05a77297-dcac-477b-95e7-489a4bd10cd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1"
Oct 31 03:45:05.986085 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.986053    5831 scope.go:122] "RemoveContainer" containerID="b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"
Oct 31 03:45:05.986403 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:05.986383    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn_odh-ai-gateway-infra(83738f85-542f-4f9b-a75c-68f6d518ebe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2"
Oct 31 03:45:05.987954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.987934    5831 scope.go:122] "RemoveContainer" containerID="632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"
Oct 31 03:45:05.988254 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:05.988231    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6_odh-ai-gateway-infra(2ae15a21-812e-420f-9a4e-2acdd757cb51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51"
Oct 31 03:45:05.989636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.989616    5831 scope.go:122] "RemoveContainer" containerID="ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"
Oct 31 03:45:05.989914 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:05.989888    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j_odh-ai-gateway-infra(0fed8bef-3530-4d58-97ed-f4d7d65da249)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249"
Oct 31 03:45:05.991407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:05.991390    5831 scope.go:122] "RemoveContainer" containerID="b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"
Oct 31 03:45:05.991675 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:05.991655    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv_odh-ai-gateway-infra(6fdb859a-40c2-492d-a0dc-d5c55bdaff69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69"
Oct 31 03:45:09.508050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:09.508006    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:45:12.991991 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:12.991949    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:45:14.134679 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:14.134645    5831 scope.go:122] "RemoveContainer" containerID="36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2"
Oct 31 03:45:15.067908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:15.067875    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerStarted","Data":"a11b4d771f6c0c1b129d5923f2216d2646e11eb890f0513a3a9d991c724da2c3"}
Oct 31 03:45:15.203556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:15.203141    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" podStartSLOduration=15.203119041 podStartE2EDuration="15.203119041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:15.188049554 +0000 UTC m=+2265.202611267" watchObservedRunningTime="2026-10-31 03:45:15.203119041 +0000 UTC m=+2265.217680750"
Oct 31 03:45:16.076997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.076959    5831 generic.go:356] "Generic (PLEG): container finished" podID="cfb10b68-777f-4856-9685-000126754651" containerID="a11b4d771f6c0c1b129d5923f2216d2646e11eb890f0513a3a9d991c724da2c3" exitCode=60
Oct 31 03:45:16.077190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.077171    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerDied","Data":"a11b4d771f6c0c1b129d5923f2216d2646e11eb890f0513a3a9d991c724da2c3"}
Oct 31 03:45:16.077250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.077215    5831 scope.go:122] "RemoveContainer" containerID="36c2ce8cd48570d0f1e818358d78d87f397f9266fc02a931b7aa3a3744745da2"
Oct 31 03:45:16.077632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.077617    5831 scope.go:122] "RemoveContainer" containerID="a11b4d771f6c0c1b129d5923f2216d2646e11eb890f0513a3a9d991c724da2c3"
Oct 31 03:45:16.077899 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:16.077882    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8_odh-ai-gateway-infra(cfb10b68-777f-4856-9685-000126754651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" podUID="cfb10b68-777f-4856-9685-000126754651"
Oct 31 03:45:16.143946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.143915    5831 scope.go:122] "RemoveContainer" containerID="632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"
Oct 31 03:45:16.144701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.144676    5831 scope.go:122] "RemoveContainer" containerID="7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"
Oct 31 03:45:16.145164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.145144    5831 scope.go:122] "RemoveContainer" containerID="ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"
Oct 31 03:45:16.229099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:16.229053    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:17.095696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.095607    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerStarted","Data":"4e517a42fada16dcc93240677364e791bd37b93c9c1c5ce7d9576dcbdf8188d0"}
Oct 31 03:45:17.098272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.098236    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerStarted","Data":"7908798f3cc1c01749dac80dbdf878ff442539151d8f76cbb7929beca4e3b93a"}
Oct 31 03:45:17.109553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.105211    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerStarted","Data":"eba5d38fd30012f26ec32a09ac70fc7c53016313acaa0224a8026299adc3085b"}
Oct 31 03:45:17.128556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.127754    5831 scope.go:122] "RemoveContainer" containerID="b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"
Oct 31 03:45:17.579890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.579860    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:17.617556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.617487    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\") pod \"cfb10b68-777f-4856-9685-000126754651\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") "
Oct 31 03:45:17.617751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.617570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\" (UniqueName: \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\") pod \"cfb10b68-777f-4856-9685-000126754651\" (UID: \"cfb10b68-777f-4856-9685-000126754651\") "
Oct 31 03:45:17.617937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.617909    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca" pod "cfb10b68-777f-4856-9685-000126754651" (UID: "cfb10b68-777f-4856-9685-000126754651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:17.618214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.618192    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb10b68-777f-4856-9685-000126754651-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:17.679393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.679355    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr" pod "cfb10b68-777f-4856-9685-000126754651" (UID: "cfb10b68-777f-4856-9685-000126754651"). InnerVolumeSpecName "kube-api-access-fgjmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:17.719505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:17.719472    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjmr\" (UniqueName: \"kubernetes.io/projected/cfb10b68-777f-4856-9685-000126754651-kube-api-access-fgjmr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.131116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.131073    5831 scope.go:122] "RemoveContainer" containerID="b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"
Oct 31 03:45:18.133178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.133138    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"
Oct 31 03:45:18.144229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.144197    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8" event={"ID":"cfb10b68-777f-4856-9685-000126754651","Type":"ContainerDied","Data":"6329cce639e8cef920282a64f19d2aab6eac61470d2a74493438305f0fdafe35"}
Oct 31 03:45:18.144687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.144668    5831 scope.go:122] "RemoveContainer" containerID="a11b4d771f6c0c1b129d5923f2216d2646e11eb890f0513a3a9d991c724da2c3"
Oct 31 03:45:18.144927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.144788    5831 generic.go:356] "Generic (PLEG): container finished" podID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerID="4e517a42fada16dcc93240677364e791bd37b93c9c1c5ce7d9576dcbdf8188d0" exitCode=60
Oct 31 03:45:18.145099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.144814    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerDied","Data":"4e517a42fada16dcc93240677364e791bd37b93c9c1c5ce7d9576dcbdf8188d0"}
Oct 31 03:45:18.145701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.145636    5831 scope.go:122] "RemoveContainer" containerID="4e517a42fada16dcc93240677364e791bd37b93c9c1c5ce7d9576dcbdf8188d0"
Oct 31 03:45:18.146009 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:18.145975    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd_odh-ai-gateway-infra(05a77297-dcac-477b-95e7-489a4bd10cd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1"
Oct 31 03:45:18.148874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.148844    5831 generic.go:356] "Generic (PLEG): container finished" podID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerID="7908798f3cc1c01749dac80dbdf878ff442539151d8f76cbb7929beca4e3b93a" exitCode=60
Oct 31 03:45:18.148986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.148944    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerDied","Data":"7908798f3cc1c01749dac80dbdf878ff442539151d8f76cbb7929beca4e3b93a"}
Oct 31 03:45:18.149838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.149814    5831 scope.go:122] "RemoveContainer" containerID="7908798f3cc1c01749dac80dbdf878ff442539151d8f76cbb7929beca4e3b93a"
Oct 31 03:45:18.150130 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:18.150098    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6_odh-ai-gateway-infra(2ae15a21-812e-420f-9a4e-2acdd757cb51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51"
Oct 31 03:45:18.158597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.158506    5831 generic.go:356] "Generic (PLEG): container finished" podID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerID="eba5d38fd30012f26ec32a09ac70fc7c53016313acaa0224a8026299adc3085b" exitCode=60
Oct 31 03:45:18.161246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.158665    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerDied","Data":"eba5d38fd30012f26ec32a09ac70fc7c53016313acaa0224a8026299adc3085b"}
Oct 31 03:45:18.161246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.159029    5831 scope.go:122] "RemoveContainer" containerID="eba5d38fd30012f26ec32a09ac70fc7c53016313acaa0224a8026299adc3085b"
Oct 31 03:45:18.161246 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:18.159303    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j_odh-ai-gateway-infra(0fed8bef-3530-4d58-97ed-f4d7d65da249)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249"
Oct 31 03:45:18.162635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.161800    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerStarted","Data":"013ade6af38c1d4916d7be4edba039f70d9d0f9628bf3c13e75b52c9ab60c754"}
Oct 31 03:45:18.170873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.170837    5831 scope.go:122] "RemoveContainer" containerID="7ec4539554062bd2371b2d8c3033d30d39a0416ee4ee63fc712b7d00359c8f72"
Oct 31 03:45:18.198218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.198191    5831 scope.go:122] "RemoveContainer" containerID="632230653f630b038eed804f1b64751ac3b79a1ea2caaf5b11c113b14e9b3c6d"
Oct 31 03:45:18.217100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.217060    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:18.227464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.226900    5831 scope.go:122] "RemoveContainer" containerID="ae47e4d5f10ad8ab4a0eea7e6e73d90c212943028a6ba46d9b998318ee3a9f84"
Oct 31 03:45:18.454758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.454717    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:18.460593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.460557    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890305-z7ll8"]
Oct 31 03:45:18.485512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.485472    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:18.584788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:18.584753    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:19.183839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.183800    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerStarted","Data":"58d4cdd411810f0c48c4e97ec438fbfad9c0cb22568290dd0277676cabc5982b"}
Oct 31 03:45:19.194567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.194516    5831 generic.go:356] "Generic (PLEG): container finished" podID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerID="013ade6af38c1d4916d7be4edba039f70d9d0f9628bf3c13e75b52c9ab60c754" exitCode=60
Oct 31 03:45:19.194701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.194582    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerDied","Data":"013ade6af38c1d4916d7be4edba039f70d9d0f9628bf3c13e75b52c9ab60c754"}
Oct 31 03:45:19.194701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.194635    5831 scope.go:122] "RemoveContainer" containerID="b4dc93faa3201af6269c30bbbf7580ed05d135b3ffae111a80d040001cb76fb8"
Oct 31 03:45:19.195431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.195330    5831 scope.go:122] "RemoveContainer" containerID="013ade6af38c1d4916d7be4edba039f70d9d0f9628bf3c13e75b52c9ab60c754"
Oct 31 03:45:19.196148 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:19.195813    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv_odh-ai-gateway-infra(6fdb859a-40c2-492d-a0dc-d5c55bdaff69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69"
Oct 31 03:45:19.343421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.343381    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:19.487545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.487389    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:19.555853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.555822    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\" (UniqueName: \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\") pod \"05a77297-dcac-477b-95e7-489a4bd10cd1\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") "
Oct 31 03:45:19.556080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.556063    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\") pod \"05a77297-dcac-477b-95e7-489a4bd10cd1\" (UID: \"05a77297-dcac-477b-95e7-489a4bd10cd1\") "
Oct 31 03:45:19.556419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.556388    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca" pod "05a77297-dcac-477b-95e7-489a4bd10cd1" (UID: "05a77297-dcac-477b-95e7-489a4bd10cd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:19.558364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.558339    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm" pod "05a77297-dcac-477b-95e7-489a4bd10cd1" (UID: "05a77297-dcac-477b-95e7-489a4bd10cd1"). InnerVolumeSpecName "kube-api-access-qfbgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:19.580677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.580644    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:19.595415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.595381    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:19.657471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.657424    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\" (UniqueName: \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\") pod \"2ae15a21-812e-420f-9a4e-2acdd757cb51\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") "
Oct 31 03:45:19.657714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.657618    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\") pod \"2ae15a21-812e-420f-9a4e-2acdd757cb51\" (UID: \"2ae15a21-812e-420f-9a4e-2acdd757cb51\") "
Oct 31 03:45:19.657806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.657730    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\") pod \"0fed8bef-3530-4d58-97ed-f4d7d65da249\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") "
Oct 31 03:45:19.657873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.657814    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\" (UniqueName: \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\") pod \"0fed8bef-3530-4d58-97ed-f4d7d65da249\" (UID: \"0fed8bef-3530-4d58-97ed-f4d7d65da249\") "
Oct 31 03:45:19.658090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658056    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca" pod "2ae15a21-812e-420f-9a4e-2acdd757cb51" (UID: "2ae15a21-812e-420f-9a4e-2acdd757cb51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:19.658180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658126    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca" pod "0fed8bef-3530-4d58-97ed-f4d7d65da249" (UID: "0fed8bef-3530-4d58-97ed-f4d7d65da249"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:19.658383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658361    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbgm\" (UniqueName: \"kubernetes.io/projected/05a77297-dcac-477b-95e7-489a4bd10cd1-kube-api-access-qfbgm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.658453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658390    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae15a21-812e-420f-9a4e-2acdd757cb51-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.658453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658406    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed8bef-3530-4d58-97ed-f4d7d65da249-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.658453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.658420    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a77297-dcac-477b-95e7-489a4bd10cd1-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.660244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.660210    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9" pod "2ae15a21-812e-420f-9a4e-2acdd757cb51" (UID: "2ae15a21-812e-420f-9a4e-2acdd757cb51"). InnerVolumeSpecName "kube-api-access-cxpm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:19.660549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.660502    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c" pod "0fed8bef-3530-4d58-97ed-f4d7d65da249" (UID: "0fed8bef-3530-4d58-97ed-f4d7d65da249"). InnerVolumeSpecName "kube-api-access-dvd9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:19.759931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.759846    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpm9\" (UniqueName: \"kubernetes.io/projected/2ae15a21-812e-420f-9a4e-2acdd757cb51-kube-api-access-cxpm9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:19.759931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:19.759891    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvd9c\" (UniqueName: \"kubernetes.io/projected/0fed8bef-3530-4d58-97ed-f4d7d65da249-kube-api-access-dvd9c\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.139799 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.139727    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb10b68-777f-4856-9685-000126754651" path="/var/lib/kubelet/pods/cfb10b68-777f-4856-9685-000126754651/volumes"
Oct 31 03:45:20.214067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.214037    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"
Oct 31 03:45:20.214827 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.214800    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd" event={"ID":"05a77297-dcac-477b-95e7-489a4bd10cd1","Type":"ContainerDied","Data":"0626a640bca0984c7c619c63a0bdf8dba07f8638f44422ec47d8dd241339fe82"}
Oct 31 03:45:20.215079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.215063    5831 scope.go:122] "RemoveContainer" containerID="4e517a42fada16dcc93240677364e791bd37b93c9c1c5ce7d9576dcbdf8188d0"
Oct 31 03:45:20.220176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.219411    5831 generic.go:356] "Generic (PLEG): container finished" podID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerID="58d4cdd411810f0c48c4e97ec438fbfad9c0cb22568290dd0277676cabc5982b" exitCode=60
Oct 31 03:45:20.220176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.219624    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerDied","Data":"58d4cdd411810f0c48c4e97ec438fbfad9c0cb22568290dd0277676cabc5982b"}
Oct 31 03:45:20.223027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.220424    5831 scope.go:122] "RemoveContainer" containerID="58d4cdd411810f0c48c4e97ec438fbfad9c0cb22568290dd0277676cabc5982b"
Oct 31 03:45:20.223027 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:20.220785    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn_odh-ai-gateway-infra(83738f85-542f-4f9b-a75c-68f6d518ebe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2"
Oct 31 03:45:20.223951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.223923    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6" event={"ID":"2ae15a21-812e-420f-9a4e-2acdd757cb51","Type":"ContainerDied","Data":"004fa55dfb4ecebd5c45c97c5037d28d353daf5db4c87c8320f68d01ec85c93c"}
Oct 31 03:45:20.224059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.224037    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"
Oct 31 03:45:20.228716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.226366    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j" event={"ID":"0fed8bef-3530-4d58-97ed-f4d7d65da249","Type":"ContainerDied","Data":"c3f7584b526fa31b2f4c76456d73d5a7c8024ec1b72b07c79ce5d1d3d2d44f27"}
Oct 31 03:45:20.228716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.226476    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"
Oct 31 03:45:20.248906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.248880    5831 scope.go:122] "RemoveContainer" containerID="b93a687d6eb1b028d7426af9989160fd6a82f2df4e5c3a58e38f8fda566e4cf6"
Oct 31 03:45:20.289644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.289614    5831 scope.go:122] "RemoveContainer" containerID="7908798f3cc1c01749dac80dbdf878ff442539151d8f76cbb7929beca4e3b93a"
Oct 31 03:45:20.306130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.306108    5831 scope.go:122] "RemoveContainer" containerID="eba5d38fd30012f26ec32a09ac70fc7c53016313acaa0224a8026299adc3085b"
Oct 31 03:45:20.429415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.429381    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:20.433430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.433405    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-14600f-29890305-qsdwd"]
Oct 31 03:45:20.445257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.445226    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:20.551179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.551112    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:20.572221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.572191    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\" (UniqueName: \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\") pod \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") "
Oct 31 03:45:20.572410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.572267    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\") pod \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\" (UID: \"6fdb859a-40c2-492d-a0dc-d5c55bdaff69\") "
Oct 31 03:45:20.572745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.572711    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca" pod "6fdb859a-40c2-492d-a0dc-d5c55bdaff69" (UID: "6fdb859a-40c2-492d-a0dc-d5c55bdaff69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:20.574924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.574890    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd" pod "6fdb859a-40c2-492d-a0dc-d5c55bdaff69" (UID: "6fdb859a-40c2-492d-a0dc-d5c55bdaff69"). InnerVolumeSpecName "kube-api-access-6kghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:20.643132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.643033    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:20.646661 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.646634    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-da5481-29890305-zqpg6"]
Oct 31 03:45:20.673442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.673411    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kghd\" (UniqueName: \"kubernetes.io/projected/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-kube-api-access-6kghd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.673442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.673440    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb859a-40c2-492d-a0dc-d5c55bdaff69-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.748880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.748442    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:20.752437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:20.752398    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a32cdb-29890305-tnj6j"]
Oct 31 03:45:21.236985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.236959    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.237004    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv" event={"ID":"6fdb859a-40c2-492d-a0dc-d5c55bdaff69","Type":"ContainerDied","Data":"a6a83382d856d1978acd9fa1c268e6394415fad6254f08dd4897af98545259ed"}
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.237062    5831 scope.go:122] "RemoveContainer" containerID="013ade6af38c1d4916d7be4edba039f70d9d0f9628bf3c13e75b52c9ab60c754"
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.393715    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.397248    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890305-fqmlv"]
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.414581    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.482620    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\") pod \"83738f85-542f-4f9b-a75c-68f6d518ebe2\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") "
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.482755    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\" (UniqueName: \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\") pod \"83738f85-542f-4f9b-a75c-68f6d518ebe2\" (UID: \"83738f85-542f-4f9b-a75c-68f6d518ebe2\") "
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.482985    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca" pod "83738f85-542f-4f9b-a75c-68f6d518ebe2" (UID: "83738f85-542f-4f9b-a75c-68f6d518ebe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.483119    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83738f85-542f-4f9b-a75c-68f6d518ebe2-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.485047    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6" pod "83738f85-542f-4f9b-a75c-68f6d518ebe2" (UID: "83738f85-542f-4f9b-a75c-68f6d518ebe2"). InnerVolumeSpecName "kube-api-access-n8gh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:21.735718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:21.596482    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gh6\" (UniqueName: \"kubernetes.io/projected/83738f85-542f-4f9b-a75c-68f6d518ebe2-kube-api-access-n8gh6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:22.086614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.086581    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": read tcp 10.132.0.2:48818->10.132.0.224:15021: read: connection reset by peer" start-of-body=
Oct 31 03:45:22.086858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.086650    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": read tcp 10.132.0.2:48818->10.132.0.224:15021: read: connection reset by peer"
Oct 31 03:45:22.137322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.137291    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" path="/var/lib/kubelet/pods/05a77297-dcac-477b-95e7-489a4bd10cd1/volumes"
Oct 31 03:45:22.137783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.137766    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" path="/var/lib/kubelet/pods/0fed8bef-3530-4d58-97ed-f4d7d65da249/volumes"
Oct 31 03:45:22.138141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.138124    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" path="/var/lib/kubelet/pods/2ae15a21-812e-420f-9a4e-2acdd757cb51/volumes"
Oct 31 03:45:22.138840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.138823    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" path="/var/lib/kubelet/pods/6fdb859a-40c2-492d-a0dc-d5c55bdaff69/volumes"
Oct 31 03:45:22.250145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.250117    5831 generic.go:356] "Generic (PLEG): container finished" podID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerID="f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40" exitCode=0
Oct 31 03:45:22.250337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.250191    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerDied","Data":"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"}
Oct 31 03:45:22.250673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.250650    5831 scope.go:122] "RemoveContainer" containerID="f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"
Oct 31 03:45:22.252071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252047    5831 generic.go:356] "Generic (PLEG): container finished" podID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerID="7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b" exitCode=0
Oct 31 03:45:22.252184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252102    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerDied","Data":"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"}
Oct 31 03:45:22.252386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252362    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.252467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252433    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.252520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252474    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.252520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.252505    5831 scope.go:122] "RemoveContainer" containerID="7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"
Oct 31 03:45:22.253776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.253757    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"
Oct 31 03:45:22.253874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.253772    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn" event={"ID":"83738f85-542f-4f9b-a75c-68f6d518ebe2","Type":"ContainerDied","Data":"546b4d2401cafb4c72fe0be11673831d3122b761ccba0fb0205911f327ec4899"}
Oct 31 03:45:22.253874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.253823    5831 scope.go:122] "RemoveContainer" containerID="58d4cdd411810f0c48c4e97ec438fbfad9c0cb22568290dd0277676cabc5982b"
Oct 31 03:45:22.254159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.254078    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.254236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.254158    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.254236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.254208    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:22.547599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.546044    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:22.547599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:22.546105    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-dab9ee-29890305-rcgzn"]
Oct 31 03:45:23.263243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.263211    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerStarted","Data":"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"}
Oct 31 03:45:23.266706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.266671    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerStarted","Data":"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"}
Oct 31 03:45:23.650632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.649626    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:45:23.650632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.649798    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:23.650632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.649839    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:23.819553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.816975    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:45:23.819553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.817109    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:23.819553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:23.817152    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:24.158285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:24.158241    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" path="/var/lib/kubelet/pods/83738f85-542f-4f9b-a75c-68f6d518ebe2/volumes"
Oct 31 03:45:24.648037 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:24.648006    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:24.648520 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:24.648494    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:24.822553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:24.819555    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:24.822553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:24.819626    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:25.045725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:25.045622    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:45:25.648296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:25.648261    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:25.648772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:25.648314    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:25.830556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:25.818634    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:25.830556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:25.818733    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:26.648481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:26.648439    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:26.649008 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:26.648494    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:26.819554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:26.818938    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:26.819554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:26.819006    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:27.647722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:27.647687    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:27.647930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:27.647777    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:27.823588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:27.822690    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:27.823588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:27.822805    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:28.647783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:28.647750    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 31 03:45:28.648027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:28.648006    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 31 03:45:28.819970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:28.819931    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 31 03:45:28.820271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:28.820238    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 31 03:45:29.654792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:29.654763    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:45:29.656047 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:29.656025    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:45:29.821657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:29.821623    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:45:29.822032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:29.822009    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:45:29.822852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:29.822831    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:45:31.664863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:31.664826    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:45:31.783616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:31.783579    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:45:34.905545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:34.905502    5831 scope.go:122] "RemoveContainer" containerID="46c579f58f9783ff5ef6464e7864f53853ac35094c0e3019a26f21785770865f"
Oct 31 03:45:36.665836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:36.665792    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" podUID="616721c3-ae66-40b2-9f49-2c5059943ea6" containerName="payload-processing" containerID="cri-o://a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69" gracePeriod=25
Oct 31 03:45:36.783970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:36.783907    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" podUID="bbfa7bcc-164a-4372-9557-a478b92b35f5" containerName="payload-pre-processing" containerID="cri-o://fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613" gracePeriod=25
Oct 31 03:45:36.951420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:36.951394    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:45:37.079487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.079462    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:45:37.109707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.109671    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\") pod \"616721c3-ae66-40b2-9f49-2c5059943ea6\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") "
Oct 31 03:45:37.109987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.109819    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\" (UniqueName: \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\") pod \"616721c3-ae66-40b2-9f49-2c5059943ea6\" (UID: \"616721c3-ae66-40b2-9f49-2c5059943ea6\") "
Oct 31 03:45:37.110232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.110202    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume" pod "616721c3-ae66-40b2-9f49-2c5059943ea6" (UID: "616721c3-ae66-40b2-9f49-2c5059943ea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:37.110601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.110579    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616721c3-ae66-40b2-9f49-2c5059943ea6-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:37.112261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.112238    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h" pod "616721c3-ae66-40b2-9f49-2c5059943ea6" (UID: "616721c3-ae66-40b2-9f49-2c5059943ea6"). InnerVolumeSpecName "kube-api-access-mgj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:37.212116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.212072    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\") pod \"bbfa7bcc-164a-4372-9557-a478b92b35f5\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") "
Oct 31 03:45:37.212327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.212221    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\" (UniqueName: \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\") pod \"bbfa7bcc-164a-4372-9557-a478b92b35f5\" (UID: \"bbfa7bcc-164a-4372-9557-a478b92b35f5\") "
Oct 31 03:45:37.212415 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.212393    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume" pod "bbfa7bcc-164a-4372-9557-a478b92b35f5" (UID: "bbfa7bcc-164a-4372-9557-a478b92b35f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:37.213084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.213006    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa7bcc-164a-4372-9557-a478b92b35f5-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:37.213084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.213036    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgj6h\" (UniqueName: \"kubernetes.io/projected/616721c3-ae66-40b2-9f49-2c5059943ea6-kube-api-access-mgj6h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:37.214757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.214729    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt" pod "bbfa7bcc-164a-4372-9557-a478b92b35f5" (UID: "bbfa7bcc-164a-4372-9557-a478b92b35f5"). InnerVolumeSpecName "kube-api-access-jkzbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:37.314080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.314039    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzbt\" (UniqueName: \"kubernetes.io/projected/bbfa7bcc-164a-4372-9557-a478b92b35f5-kube-api-access-jkzbt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:37.450970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.450935    5831 generic.go:356] "Generic (PLEG): container finished" podID="616721c3-ae66-40b2-9f49-2c5059943ea6" containerID="a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69" exitCode=0
Oct 31 03:45:37.451166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.450995    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"
Oct 31 03:45:37.451166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.451016    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" event={"ID":"616721c3-ae66-40b2-9f49-2c5059943ea6","Type":"ContainerDied","Data":"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"}
Oct 31 03:45:37.451166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.451090    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp" event={"ID":"616721c3-ae66-40b2-9f49-2c5059943ea6","Type":"ContainerDied","Data":"0bb9ce2e66c0f48dfa936783fc3c3582f5752e385b35f33c2c4bc9f3d9c3f451"}
Oct 31 03:45:37.451166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.451111    5831 scope.go:122] "RemoveContainer" containerID="a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"
Oct 31 03:45:37.452725 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.452698    5831 generic.go:356] "Generic (PLEG): container finished" podID="bbfa7bcc-164a-4372-9557-a478b92b35f5" containerID="fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613" exitCode=0
Oct 31 03:45:37.452855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.452748    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"
Oct 31 03:45:37.452855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.452752    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" event={"ID":"bbfa7bcc-164a-4372-9557-a478b92b35f5","Type":"ContainerDied","Data":"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"}
Oct 31 03:45:37.452855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.452791    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk" event={"ID":"bbfa7bcc-164a-4372-9557-a478b92b35f5","Type":"ContainerDied","Data":"3276d26072c390a3193aecb2a7ebc9b0f74049f9d26ed059d0a5da33127d21bd"}
Oct 31 03:45:37.462689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.462674    5831 scope.go:122] "RemoveContainer" containerID="a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"
Oct 31 03:45:37.462982 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:37.462964    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69\": container with ID starting with a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69 not found: ID does not exist" containerID="a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"
Oct 31 03:45:37.463036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.462991    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69"} err="failed to get container status \"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69\": rpc error: code = NotFound desc = could not find container \"a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69\": container with ID starting with a0349acb193f26576541315d3af2a1bd13c069bf7bb2bdf3a9154b717a504d69 not found: ID does not exist"
Oct 31 03:45:37.463036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.463007    5831 scope.go:122] "RemoveContainer" containerID="fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"
Oct 31 03:45:37.472981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.472962    5831 scope.go:122] "RemoveContainer" containerID="fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"
Oct 31 03:45:37.473241 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:37.473222    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613\": container with ID starting with fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613 not found: ID does not exist" containerID="fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"
Oct 31 03:45:37.473290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.473248    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613"} err="failed to get container status \"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613\": rpc error: code = NotFound desc = could not find container \"fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613\": container with ID starting with fb5aa7bcbbecb5cde4658e4ed5b22ed641320f692f48d5548ee9ba292f916613 not found: ID does not exist"
Oct 31 03:45:37.603882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.603845    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:45:37.650026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.649996    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-da5481-84569569df-g4kvp"]
Oct 31 03:45:37.705095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.705063    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:45:37.711652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:37.711623    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-da5481-58cf8d7c8c-p4rbk"]
Oct 31 03:45:38.136924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:38.136883    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616721c3-ae66-40b2-9f49-2c5059943ea6" path="/var/lib/kubelet/pods/616721c3-ae66-40b2-9f49-2c5059943ea6/volumes"
Oct 31 03:45:38.137400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:38.137378    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfa7bcc-164a-4372-9557-a478b92b35f5" path="/var/lib/kubelet/pods/bbfa7bcc-164a-4372-9557-a478b92b35f5/volumes"
Oct 31 03:45:42.979758 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:45:42.979716    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:45:46.497386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:46.497344    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:45:54.157599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:54.157569    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq_288aec47-ab90-45a2-a58e-90bccd98f0cd/payload-processing/0.log"
Oct 31 03:45:54.889558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:54.889508    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb_a325465a-6593-4466-b0a4-e260b56e298e/payload-processing/0.log"
Oct 31 03:45:55.115056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.115023    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb_a325465a-6593-4466-b0a4-e260b56e298e/payload-processing/0.log"
Oct 31 03:45:55.491137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.491101    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:45:55.492117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492091    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492118    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492140    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492148    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492162    5831 cpu_manager.go:414] "containerMap: removing container" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492169    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492186    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492193    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492208    5831 cpu_manager.go:414] "containerMap: removing container" podUID="616721c3-ae66-40b2-9f49-2c5059943ea6" containerName="payload-processing"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492216    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616721c3-ae66-40b2-9f49-2c5059943ea6" containerName="payload-processing"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492228    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492236    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492244    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492251    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492261    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492268    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492282    5831 cpu_manager.go:414] "containerMap: removing container" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492289    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492298    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492304    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492322    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492328    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492339    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492345    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492358    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492366    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492376    5831 cpu_manager.go:414] "containerMap: removing container" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492383    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492402    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492409    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492421    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bbfa7bcc-164a-4372-9557-a478b92b35f5" containerName="payload-pre-processing"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492428    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfa7bcc-164a-4372-9557-a478b92b35f5" containerName="payload-pre-processing"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492463    5831 cpu_manager.go:414] "containerMap: removing container" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492469    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492663    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfa7bcc-164a-4372-9557-a478b92b35f5" containerName="payload-pre-processing"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492682    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492693    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492700    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492710    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492719    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492728    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492737    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492744    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="616721c3-ae66-40b2-9f49-2c5059943ea6" containerName="payload-processing"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492754    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492761    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492770    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdb859a-40c2-492d-a0dc-d5c55bdaff69" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492778    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492787    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492796    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492804    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.492905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.492814    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:45:55.497327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.497306    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.593982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.593946    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fc85c7-openshift-default-dockercfg-6l8wz\""
Oct 31 03:45:55.596123 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.596074    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.596590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.596571    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.596917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.596856    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.597078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.597064    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.597301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.597286    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.597497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.597483    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.597837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.597723    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.601625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.601598    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hct\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.601820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.601801    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.604758 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.604737    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:45:55.708103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.708069    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.709075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.709050    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.709615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.709596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.709795 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.709778    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.709975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.709962    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.710114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.710100    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.710252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.710239    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.710354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.710342    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hct\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.710466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.710453    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.735629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.708899    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.735629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.709498    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.743277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.743149    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.743832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.743807    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.744196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.744160    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.746162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.746139    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.746904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.746870    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.802705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.802671    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.803084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.803063    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hct\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\") pod \"e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") " pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:55.861006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:55.860973    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:56.295102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.295067    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:45:56.296012 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:45:56.295988    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c4e804_4847_4652_b2a6_23f68c2fe550.slice/crio-8961dd480cb2b1a83d224cc2745b4f0e1d7c4490cd970ac6f4ef760375d31674 WatchSource:0}: Error finding container 8961dd480cb2b1a83d224cc2745b4f0e1d7c4490cd970ac6f4ef760375d31674: Status 404 returned error can't find the container with id 8961dd480cb2b1a83d224cc2745b4f0e1d7c4490cd970ac6f4ef760375d31674
Oct 31 03:45:56.298664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.298629    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:56.298768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.298698    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:56.298768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.298729    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:45:56.651628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.651490    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerStarted","Data":"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"}
Oct 31 03:45:56.651628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.651613    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerStarted","Data":"8961dd480cb2b1a83d224cc2745b4f0e1d7c4490cd970ac6f4ef760375d31674"}
Oct 31 03:45:56.764261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.764198    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podStartSLOduration=1.7641780059999999 podStartE2EDuration="1.764178006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:56.759913873 +0000 UTC m=+2306.774475581" watchObservedRunningTime="2026-10-31 03:45:56.764178006 +0000 UTC m=+2306.778739716"
Oct 31 03:45:56.861610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:56.861577    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:57.867713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:57.867674    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:58.666666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:58.666631    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:45:58.667820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:45:58.667803    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:46:06.566039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:06.565997    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:46:06.716401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:06.716367    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:46:11.567351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.567301    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" podUID="00df6b23-b876-401f-9f15-fa8fd6cc980a" containerName="payload-processing" containerID="cri-o://6919f870ee674a6db41f1c2196a8a48a65ad1718f507f08a999e77d05c4e6775" gracePeriod=25
Oct 31 03:46:11.717129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.717025    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" podUID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" containerName="payload-pre-processing" containerID="cri-o://d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf" gracePeriod=25
Oct 31 03:46:11.776069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.776037    5831 generic.go:356] "Generic (PLEG): container finished" podID="00df6b23-b876-401f-9f15-fa8fd6cc980a" containerID="6919f870ee674a6db41f1c2196a8a48a65ad1718f507f08a999e77d05c4e6775" exitCode=0
Oct 31 03:46:11.776262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.776106    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" event={"ID":"00df6b23-b876-401f-9f15-fa8fd6cc980a","Type":"ContainerDied","Data":"6919f870ee674a6db41f1c2196a8a48a65ad1718f507f08a999e77d05c4e6775"}
Oct 31 03:46:11.872203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.872169    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:46:11.972987 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.972949    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\" (UniqueName: \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\") pod \"00df6b23-b876-401f-9f15-fa8fd6cc980a\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") "
Oct 31 03:46:11.973183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.973031    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\") pod \"00df6b23-b876-401f-9f15-fa8fd6cc980a\" (UID: \"00df6b23-b876-401f-9f15-fa8fd6cc980a\") "
Oct 31 03:46:11.973442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.973410    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume" pod "00df6b23-b876-401f-9f15-fa8fd6cc980a" (UID: "00df6b23-b876-401f-9f15-fa8fd6cc980a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:11.973935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.973913    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00df6b23-b876-401f-9f15-fa8fd6cc980a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:11.975377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:11.975344    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2" pod "00df6b23-b876-401f-9f15-fa8fd6cc980a" (UID: "00df6b23-b876-401f-9f15-fa8fd6cc980a"). InnerVolumeSpecName "kube-api-access-lnqv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:12.011582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.011554    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:46:12.035744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.035711    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:46:12.074593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.074561    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\") pod \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") "
Oct 31 03:46:12.074744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.074678    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\" (UniqueName: \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\") pod \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\" (UID: \"1ef5909c-2322-4be3-a4b2-d74a76aab0ad\") "
Oct 31 03:46:12.074880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.074857    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume" pod "1ef5909c-2322-4be3-a4b2-d74a76aab0ad" (UID: "1ef5909c-2322-4be3-a4b2-d74a76aab0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:12.075125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.075103    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnqv2\" (UniqueName: \"kubernetes.io/projected/00df6b23-b876-401f-9f15-fa8fd6cc980a-kube-api-access-lnqv2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:12.075125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.075120    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:12.077114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.077089    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8" pod "1ef5909c-2322-4be3-a4b2-d74a76aab0ad" (UID: "1ef5909c-2322-4be3-a4b2-d74a76aab0ad"). InnerVolumeSpecName "kube-api-access-6pmx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:12.176567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.176486    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pmx8\" (UniqueName: \"kubernetes.io/projected/1ef5909c-2322-4be3-a4b2-d74a76aab0ad-kube-api-access-6pmx8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:12.196705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.196658    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:46:12.784907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.784873    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"
Oct 31 03:46:12.785467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.784970    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb" event={"ID":"00df6b23-b876-401f-9f15-fa8fd6cc980a","Type":"ContainerDied","Data":"f84b54050406fb5735b5424be50ce64299477449b2e7b2fd6f8f676f34cdbde1"}
Oct 31 03:46:12.785467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.785028    5831 scope.go:122] "RemoveContainer" containerID="6919f870ee674a6db41f1c2196a8a48a65ad1718f507f08a999e77d05c4e6775"
Oct 31 03:46:12.787212 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.787186    5831 generic.go:356] "Generic (PLEG): container finished" podID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" containerID="d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf" exitCode=0
Oct 31 03:46:12.787344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.787318    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" event={"ID":"1ef5909c-2322-4be3-a4b2-d74a76aab0ad","Type":"ContainerDied","Data":"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"}
Oct 31 03:46:12.787425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.787396    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8" event={"ID":"1ef5909c-2322-4be3-a4b2-d74a76aab0ad","Type":"ContainerDied","Data":"73c3fe1b5aded78232d384477f7d0c11be7e3bd6f90c11b892068441da2b7736"}
Oct 31 03:46:12.787491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.787279    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"
Oct 31 03:46:12.797599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.797583    5831 scope.go:122] "RemoveContainer" containerID="d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"
Oct 31 03:46:12.807562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.807508    5831 scope.go:122] "RemoveContainer" containerID="d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"
Oct 31 03:46:12.807898 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:12.807877    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf\": container with ID starting with d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf not found: ID does not exist" containerID="d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"
Oct 31 03:46:12.807985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.807908    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf"} err="failed to get container status \"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf\": rpc error: code = NotFound desc = could not find container \"d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf\": container with ID starting with d6bbaa47fb3def6e39e4f7dfba88257410ab1a156e22ff844584b435daa9adcf not found: ID does not exist"
Oct 31 03:46:12.941370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.938967    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:46:12.943243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:12.943210    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6f9fd2-648ddf94c-l2lj8"]
Oct 31 03:46:12.979676 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:12.979637    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:46:13.045718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:13.045674    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:46:13.051064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:13.051036    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6f9fd2-7cd564cf57-zd5vb"]
Oct 31 03:46:14.135971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:14.135936    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00df6b23-b876-401f-9f15-fa8fd6cc980a" path="/var/lib/kubelet/pods/00df6b23-b876-401f-9f15-fa8fd6cc980a/volumes"
Oct 31 03:46:14.136511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:14.136495    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" path="/var/lib/kubelet/pods/1ef5909c-2322-4be3-a4b2-d74a76aab0ad/volumes"
Oct 31 03:46:16.817426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:16.817386    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:46:16.818312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:16.818275    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" podUID="4fec983c-d1d4-44f5-8980-0125058887ae" containerName="maas-api" containerID="cri-o://aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d" gracePeriod=30
Oct 31 03:46:17.036595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.036551    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" podUID="a1536fbf-903f-47e8-8b35-fd283c22a34b" containerName="payload-processing" containerID="cri-o://943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94" gracePeriod=25
Oct 31 03:46:17.190504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.190477    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:46:17.206556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.200646    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" podUID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" containerName="payload-pre-processing" containerID="cri-o://5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a" gracePeriod=25
Oct 31 03:46:17.269588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.267164    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") pod \"4fec983c-d1d4-44f5-8980-0125058887ae\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") "
Oct 31 03:46:17.269588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.267230    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\") pod \"4fec983c-d1d4-44f5-8980-0125058887ae\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") "
Oct 31 03:46:17.269588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.267343    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\") pod \"4fec983c-d1d4-44f5-8980-0125058887ae\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") "
Oct 31 03:46:17.269588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.267545    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\" (UniqueName: \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\") pod \"4fec983c-d1d4-44f5-8980-0125058887ae\" (UID: \"4fec983c-d1d4-44f5-8980-0125058887ae\") "
Oct 31 03:46:17.273773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.273575    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca" pod "4fec983c-d1d4-44f5-8980-0125058887ae" (UID: "4fec983c-d1d4-44f5-8980-0125058887ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:17.279495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.279439    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls" pod "4fec983c-d1d4-44f5-8980-0125058887ae" (UID: "4fec983c-d1d4-44f5-8980-0125058887ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:17.280078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.280048    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs" pod "4fec983c-d1d4-44f5-8980-0125058887ae" (UID: "4fec983c-d1d4-44f5-8980-0125058887ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:17.280281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.280098    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb" pod "4fec983c-d1d4-44f5-8980-0125058887ae" (UID: "4fec983c-d1d4-44f5-8980-0125058887ae"). InnerVolumeSpecName "kube-api-access-wz2vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:17.369644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.369505    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.369644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.369568    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fec983c-d1d4-44f5-8980-0125058887ae-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.369644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.369588    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fec983c-d1d4-44f5-8980-0125058887ae-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.369644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.369607    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz2vb\" (UniqueName: \"kubernetes.io/projected/4fec983c-d1d4-44f5-8980-0125058887ae-kube-api-access-wz2vb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.462363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.462293    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:46:17.531151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.531127    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:46:17.571863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.571831    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\") pod \"a1536fbf-903f-47e8-8b35-fd283c22a34b\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") "
Oct 31 03:46:17.572064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.571905    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\" (UniqueName: \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\") pod \"a1536fbf-903f-47e8-8b35-fd283c22a34b\" (UID: \"a1536fbf-903f-47e8-8b35-fd283c22a34b\") "
Oct 31 03:46:17.572329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.572297    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume" pod "a1536fbf-903f-47e8-8b35-fd283c22a34b" (UID: "a1536fbf-903f-47e8-8b35-fd283c22a34b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:17.574595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.574557    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw" pod "a1536fbf-903f-47e8-8b35-fd283c22a34b" (UID: "a1536fbf-903f-47e8-8b35-fd283c22a34b"). InnerVolumeSpecName "kube-api-access-shqvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:17.673331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.673245    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\") pod \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") "
Oct 31 03:46:17.673331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.673300    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\" (UniqueName: \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\") pod \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\" (UID: \"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077\") "
Oct 31 03:46:17.673809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.673780    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume" pod "59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" (UID: "59fe3b91-96e1-4dcb-b1f4-9abdb0f61077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:17.673909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.673857    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1536fbf-903f-47e8-8b35-fd283c22a34b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.673909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.673879    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqvw\" (UniqueName: \"kubernetes.io/projected/a1536fbf-903f-47e8-8b35-fd283c22a34b-kube-api-access-shqvw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.676644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.676614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w" pod "59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" (UID: "59fe3b91-96e1-4dcb-b1f4-9abdb0f61077"). InnerVolumeSpecName "kube-api-access-hgr6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:17.775116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.775077    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.775116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.775109    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgr6w\" (UniqueName: \"kubernetes.io/projected/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077-kube-api-access-hgr6w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:17.847580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.847502    5831 generic.go:356] "Generic (PLEG): container finished" podID="4fec983c-d1d4-44f5-8980-0125058887ae" containerID="aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d" exitCode=0
Oct 31 03:46:17.848355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.847754    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" event={"ID":"4fec983c-d1d4-44f5-8980-0125058887ae","Type":"ContainerDied","Data":"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"}
Oct 31 03:46:17.848355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.847823    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf" event={"ID":"4fec983c-d1d4-44f5-8980-0125058887ae","Type":"ContainerDied","Data":"8f9b064c155093b2199f2d44303d8ee1394dc0d0db8e09a9d5b7e904c164e86c"}
Oct 31 03:46:17.848355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.847870    5831 scope.go:122] "RemoveContainer" containerID="aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"
Oct 31 03:46:17.848355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.848062    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"
Oct 31 03:46:17.893769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.893720    5831 generic.go:356] "Generic (PLEG): container finished" podID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" containerID="5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a" exitCode=0
Oct 31 03:46:17.907245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.907214    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"
Oct 31 03:46:17.928960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.907600    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" event={"ID":"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077","Type":"ContainerDied","Data":"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"}
Oct 31 03:46:17.928960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.907674    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c" event={"ID":"59fe3b91-96e1-4dcb-b1f4-9abdb0f61077","Type":"ContainerDied","Data":"30598c53a64c920ea194075d03a814881665c944d759c095331da9a20ca7ff43"}
Oct 31 03:46:17.928960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.920483    5831 scope.go:122] "RemoveContainer" containerID="aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"
Oct 31 03:46:17.942561 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:17.942501    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d\": container with ID starting with aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d not found: ID does not exist" containerID="aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"
Oct 31 03:46:17.942742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.942569    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d"} err="failed to get container status \"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d\": rpc error: code = NotFound desc = could not find container \"aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d\": container with ID starting with aea5c111ee461e3237d3c862414de9152a8b04973a67eb2fcf4a62f3b6c7ae2d not found: ID does not exist"
Oct 31 03:46:17.942742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.942594    5831 scope.go:122] "RemoveContainer" containerID="5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"
Oct 31 03:46:17.956734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.951919    5831 generic.go:356] "Generic (PLEG): container finished" podID="a1536fbf-903f-47e8-8b35-fd283c22a34b" containerID="943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94" exitCode=0
Oct 31 03:46:17.956734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.951973    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" event={"ID":"a1536fbf-903f-47e8-8b35-fd283c22a34b","Type":"ContainerDied","Data":"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"}
Oct 31 03:46:17.956734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.952018    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68" event={"ID":"a1536fbf-903f-47e8-8b35-fd283c22a34b","Type":"ContainerDied","Data":"c5d51d4224190862b3afd2834057fda7250ae646af81418c81c1817f9b421d62"}
Oct 31 03:46:17.956734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.952107    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"
Oct 31 03:46:17.982699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.982670    5831 scope.go:122] "RemoveContainer" containerID="5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"
Oct 31 03:46:17.983081 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:17.983055    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a\": container with ID starting with 5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a not found: ID does not exist" containerID="5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"
Oct 31 03:46:17.983179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.983089    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a"} err="failed to get container status \"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a\": rpc error: code = NotFound desc = could not find container \"5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a\": container with ID starting with 5988d2eeecb61f9ee3b689475b831fa382687e84c72a4a79523674f00f76c05a not found: ID does not exist"
Oct 31 03:46:17.983179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.983110    5831 scope.go:122] "RemoveContainer" containerID="943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"
Oct 31 03:46:17.995739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.994935    5831 scope.go:122] "RemoveContainer" containerID="943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"
Oct 31 03:46:17.995739 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:17.995286    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94\": container with ID starting with 943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94 not found: ID does not exist" containerID="943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"
Oct 31 03:46:17.995739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:17.995315    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94"} err="failed to get container status \"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94\": rpc error: code = NotFound desc = could not find container \"943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94\": container with ID starting with 943712c286de39820c1f3b982272241bc616424e473272c825716dcb9c619e94 not found: ID does not exist"
Oct 31 03:46:18.137074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.137046    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:46:18.137247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.137085    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-8678955c47-dnf7c"]
Oct 31 03:46:18.237595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.237508    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:46:18.245510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.245479    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f4b947575-g4l68"]
Oct 31 03:46:18.346328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.346290    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:46:18.349805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:18.349781    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-da5481-c757bcbdf-mtcrf"]
Oct 31 03:46:20.144670 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:20.144626    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fec983c-d1d4-44f5-8980-0125058887ae" path="/var/lib/kubelet/pods/4fec983c-d1d4-44f5-8980-0125058887ae/volumes"
Oct 31 03:46:20.145115 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:20.145099    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" path="/var/lib/kubelet/pods/59fe3b91-96e1-4dcb-b1f4-9abdb0f61077/volumes"
Oct 31 03:46:20.145507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:20.145493    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1536fbf-903f-47e8-8b35-fd283c22a34b" path="/var/lib/kubelet/pods/a1536fbf-903f-47e8-8b35-fd283c22a34b/volumes"
Oct 31 03:46:29.197675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:29.197585    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:46:29.198289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:29.197912    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy" containerID="cri-o://35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f" gracePeriod=30
Oct 31 03:46:34.480300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.480230    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:46:34.502373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502340    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502378    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502414    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502459    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502504    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502557    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502797    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502801    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502851    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.502891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.502882    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\") pod \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\" (UID: \"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824\") "
Oct 31 03:46:34.503426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503383    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:34.503610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503591    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.503702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503620    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.503702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503590    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:34.503702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503639    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.503702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.503654    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.504999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.504969    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:46:34.505434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.505411    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "kube-api-access-bwplk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:34.505517 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.505430    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:34.505818 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.505801    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token" pod "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" (UID: "ead5ac1b-c6b4-4ea7-9bc2-8423134fa824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:34.605280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.605239    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.605280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.605271    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.605280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.605285    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.605622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.605299    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.605622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.605313    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwplk\" (UniqueName: \"kubernetes.io/projected/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824-kube-api-access-bwplk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:34.962171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.959600    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:46:34.977199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:34.977158    5831 scope.go:122] "RemoveContainer" containerID="c06d3a8a5174abb16e9358d3b7692256923aa024ae48b5b6fe6607769543477a"
Oct 31 03:46:35.094951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.094921    5831 generic.go:356] "Generic (PLEG): container finished" podID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerID="35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f" exitCode=0
Oct 31 03:46:35.095129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.094988    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"
Oct 31 03:46:35.095129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.095010    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerDied","Data":"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"}
Oct 31 03:46:35.095129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.095062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6" event={"ID":"ead5ac1b-c6b4-4ea7-9bc2-8423134fa824","Type":"ContainerDied","Data":"d7913c14a9eb99e831cfa574935c8c5cbce741a83640ab34ce34c3ef521eb0b9"}
Oct 31 03:46:35.095129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.095093    5831 scope.go:122] "RemoveContainer" containerID="35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"
Oct 31 03:46:35.107777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.107755    5831 scope.go:122] "RemoveContainer" containerID="35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"
Oct 31 03:46:35.108090 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:35.108069    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f\": container with ID starting with 35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f not found: ID does not exist" containerID="35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"
Oct 31 03:46:35.108139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.108100    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f"} err="failed to get container status \"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f\": rpc error: code = NotFound desc = could not find container \"35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f\": container with ID starting with 35efe770dd71b8bd5a66eaad2f168198b83971c63f1c4db1660c844259ae7e1f not found: ID does not exist"
Oct 31 03:46:35.308469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.308367    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:46:35.311686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:35.311660    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-da5481-openshift-default-84c5f4d769-bfzl6"]
Oct 31 03:46:36.135964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:36.135928    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" path="/var/lib/kubelet/pods/ead5ac1b-c6b4-4ea7-9bc2-8423134fa824/volumes"
Oct 31 03:46:37.286445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.286403    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"]
Oct 31 03:46:37.287272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287249    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287274    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287287    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" containerName="payload-pre-processing"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287294    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" containerName="payload-pre-processing"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287309    5831 cpu_manager.go:414] "containerMap: removing container" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287316    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83738f85-542f-4f9b-a75c-68f6d518ebe2" containerName="cleanup"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287339    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a1536fbf-903f-47e8-8b35-fd283c22a34b" containerName="payload-processing"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287346    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1536fbf-903f-47e8-8b35-fd283c22a34b" containerName="payload-processing"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287368    5831 cpu_manager.go:414] "containerMap: removing container" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287374    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a77297-dcac-477b-95e7-489a4bd10cd1" containerName="cleanup"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287395    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:46:37.287403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287402    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287419    5831 cpu_manager.go:414] "containerMap: removing container" podUID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" containerName="payload-pre-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287427    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" containerName="payload-pre-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287450    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4fec983c-d1d4-44f5-8980-0125058887ae" containerName="maas-api"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287457    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fec983c-d1d4-44f5-8980-0125058887ae" containerName="maas-api"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287482    5831 cpu_manager.go:414] "containerMap: removing container" podUID="00df6b23-b876-401f-9f15-fa8fd6cc980a" containerName="payload-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287491    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00df6b23-b876-401f-9f15-fa8fd6cc980a" containerName="payload-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287671    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fe3b91-96e1-4dcb-b1f4-9abdb0f61077" containerName="payload-pre-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287689    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fec983c-d1d4-44f5-8980-0125058887ae" containerName="maas-api"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287700    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed8bef-3530-4d58-97ed-f4d7d65da249" containerName="cleanup"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287709    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="00df6b23-b876-401f-9f15-fa8fd6cc980a" containerName="payload-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287722    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287731    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef5909c-2322-4be3-a4b2-d74a76aab0ad" containerName="payload-pre-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287743    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb10b68-777f-4856-9685-000126754651" containerName="cleanup"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287752    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1536fbf-903f-47e8-8b35-fd283c22a34b" containerName="payload-processing"
Oct 31 03:46:37.288001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.287759    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae15a21-812e-420f-9a4e-2acdd757cb51" containerName="cleanup"
Oct 31 03:46:37.293205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.293181    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.340275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.340244    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.340454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.340327    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgzp\" (UniqueName: \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.340513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.340467    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.343353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.343326    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"]
Oct 31 03:46:37.384547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.384497    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:46:37.384735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.384576    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:46:37.442182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.442143    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.442379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.442220    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgzp\" (UniqueName: \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.442379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.442279    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.443040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.443022    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.444916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.444895    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.503679 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.503646    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgzp\" (UniqueName: \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\") pod \"maas-api-revoke-sub-bfc088a5-s4l2d\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:37.607910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:37.607818    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:38.011656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:38.011634    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"]
Oct 31 03:46:38.013980 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:46:38.013951    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f05d7a_d8c9_40cd_bc38_0f4db8d95241.slice/crio-8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0 WatchSource:0}: Error finding container 8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0: Status 404 returned error can't find the container with id 8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0
Oct 31 03:46:38.120984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:38.120943    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d" event={"ID":"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241","Type":"ContainerStarted","Data":"8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0"}
Oct 31 03:46:39.130696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:39.130667    5831 generic.go:356] "Generic (PLEG): container finished" podID="e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" containerID="13daff079fc0bef4b8f264782e111f6654db75865f7683b298c8a87b919789a2" exitCode=0
Oct 31 03:46:39.131081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:39.130790    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d" event={"ID":"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241","Type":"ContainerDied","Data":"13daff079fc0bef4b8f264782e111f6654db75865f7683b298c8a87b919789a2"}
Oct 31 03:46:40.544552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.543211    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:40.644050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.644018    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\") pod \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") "
Oct 31 03:46:40.644208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.644164    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\") pod \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") "
Oct 31 03:46:40.644290 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.644268    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\" (UniqueName: \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\") pod \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\" (UID: \"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241\") "
Oct 31 03:46:40.644390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.644364    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca" pod "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" (UID: "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:40.644770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.644745    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.646509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.646488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca" pod "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" (UID: "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:40.646603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.646560    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp" pod "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" (UID: "e3f05d7a-d8c9-40cd-bc38-0f4db8d95241"). InnerVolumeSpecName "kube-api-access-slgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:40.746414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.746372    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:40.746414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:40.746412    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgzp\" (UniqueName: \"kubernetes.io/projected/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241-kube-api-access-slgzp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:41.152588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:41.152510    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"
Oct 31 03:46:41.152781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:41.152590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d" event={"ID":"e3f05d7a-d8c9-40cd-bc38-0f4db8d95241","Type":"ContainerDied","Data":"8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0"}
Oct 31 03:46:41.152781 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:41.152644    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a714660cf7400fc21d2d62015ae5979b9b959526eb79052584224e3f2784cb0"
Oct 31 03:46:42.992562 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:42.992489    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:46:45.258999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:45.258961    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:46:45.259607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:45.259167    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy" containerID="cri-o://327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31" gracePeriod=30
Oct 31 03:46:50.537924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.537860    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:46:50.683353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683310    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683394    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683497    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683625    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683690    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683727    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:50.683832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683764    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.683832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683812    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.684160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683863    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\") pod \"a052139c-cb90-4787-85b2-63bac51e0aed\" (UID: \"a052139c-cb90-4787-85b2-63bac51e0aed\") "
Oct 31 03:46:50.684160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.683958    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:50.684160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684064    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:50.684341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684243    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:50.684467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684442    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.684873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684473    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.684873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684487    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a052139c-cb90-4787-85b2-63bac51e0aed-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.684873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684500    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.684873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.684811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:50.686867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.686830    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:50.686983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.686939    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "kube-api-access-qqj7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:50.687061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.687008    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:46:50.687129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.687106    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy" pod "a052139c-cb90-4787-85b2-63bac51e0aed" (UID: "a052139c-cb90-4787-85b2-63bac51e0aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:46:50.786169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.786130    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.786169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.786168    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqj7r\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-kube-api-access-qqj7r\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.786371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.786183    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a052139c-cb90-4787-85b2-63bac51e0aed-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.786371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.786196    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a052139c-cb90-4787-85b2-63bac51e0aed-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.786371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:50.786226    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a052139c-cb90-4787-85b2-63bac51e0aed-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:51.240927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.240886    5831 generic.go:356] "Generic (PLEG): container finished" podID="a052139c-cb90-4787-85b2-63bac51e0aed" containerID="327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31" exitCode=0
Oct 31 03:46:51.241132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.240939    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"
Oct 31 03:46:51.241132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.240963    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerDied","Data":"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"}
Oct 31 03:46:51.241132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.241018    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht" event={"ID":"a052139c-cb90-4787-85b2-63bac51e0aed","Type":"ContainerDied","Data":"d9ac467a4fe99b38736b17cfcc2c929e942964ca5b6bed650bceed4cc5fdda98"}
Oct 31 03:46:51.241132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.241040    5831 scope.go:122] "RemoveContainer" containerID="327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"
Oct 31 03:46:51.253027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.253010    5831 scope.go:122] "RemoveContainer" containerID="db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"
Oct 31 03:46:51.263285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.263264    5831 scope.go:122] "RemoveContainer" containerID="327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"
Oct 31 03:46:51.263583 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:51.263559    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31\": container with ID starting with 327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31 not found: ID does not exist" containerID="327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"
Oct 31 03:46:51.263658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.263594    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31"} err="failed to get container status \"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31\": rpc error: code = NotFound desc = could not find container \"327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31\": container with ID starting with 327f0dd99ed3fc5db7760d66e397b8facdb172d78a34a6e54d15af6476171e31 not found: ID does not exist"
Oct 31 03:46:51.263658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.263610    5831 scope.go:122] "RemoveContainer" containerID="db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"
Oct 31 03:46:51.263868 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:51.263849    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264\": container with ID starting with db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264 not found: ID does not exist" containerID="db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"
Oct 31 03:46:51.263923 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.263876    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264"} err="failed to get container status \"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264\": rpc error: code = NotFound desc = could not find container \"db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264\": container with ID starting with db9630414307e2b40e8fd888a92f4056a5a1e218fbf8d8640ba5fdaf657b2264 not found: ID does not exist"
Oct 31 03:46:51.439778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.439740    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:46:51.485898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:51.485866    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6f9fd2-openshift-default-7cf55db6b5-62pht"]
Oct 31 03:46:52.136973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:52.136941    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" path="/var/lib/kubelet/pods/a052139c-cb90-4787-85b2-63bac51e0aed/volumes"
Oct 31 03:46:52.944769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:52.944731    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:46:57.945347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:57.945302    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" podUID="8b68dc18-e915-4c58-9693-42796ac20fa5" containerName="payload-pre-processing" containerID="cri-o://3b9efe4c8071ab35dcf8553281638ff9a48cb0fd329563a0b07f13a30811a049" gracePeriod=25
Oct 31 03:46:58.314226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.314186    5831 generic.go:356] "Generic (PLEG): container finished" podID="8b68dc18-e915-4c58-9693-42796ac20fa5" containerID="3b9efe4c8071ab35dcf8553281638ff9a48cb0fd329563a0b07f13a30811a049" exitCode=0
Oct 31 03:46:58.314371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.314266    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" event={"ID":"8b68dc18-e915-4c58-9693-42796ac20fa5","Type":"ContainerDied","Data":"3b9efe4c8071ab35dcf8553281638ff9a48cb0fd329563a0b07f13a30811a049"}
Oct 31 03:46:58.314426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.314383    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh" event={"ID":"8b68dc18-e915-4c58-9693-42796ac20fa5","Type":"ContainerDied","Data":"f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe"}
Oct 31 03:46:58.314426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.314415    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f982a8c311f1daa24a5a24291a5add049bfcf6cd532128ab4eb611b2d1b00dfe"
Oct 31 03:46:58.332105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.332082    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:46:58.436195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.436162    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\") pod \"8b68dc18-e915-4c58-9693-42796ac20fa5\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") "
Oct 31 03:46:58.436383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.436204    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\" (UniqueName: \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\") pod \"8b68dc18-e915-4c58-9693-42796ac20fa5\" (UID: \"8b68dc18-e915-4c58-9693-42796ac20fa5\") "
Oct 31 03:46:58.436668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.436643    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume" pod "8b68dc18-e915-4c58-9693-42796ac20fa5" (UID: "8b68dc18-e915-4c58-9693-42796ac20fa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:58.436904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.436879    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b68dc18-e915-4c58-9693-42796ac20fa5-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.439276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.439246    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs" pod "8b68dc18-e915-4c58-9693-42796ac20fa5" (UID: "8b68dc18-e915-4c58-9693-42796ac20fa5"). InnerVolumeSpecName "kube-api-access-smmgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:58.537851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.537816    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmgs\" (UniqueName: \"kubernetes.io/projected/8b68dc18-e915-4c58-9693-42796ac20fa5-kube-api-access-smmgs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:58.848899 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.848849    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:46:58.849127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.849082    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" podUID="288aec47-ab90-45a2-a58e-90bccd98f0cd" containerName="payload-processing" containerID="cri-o://5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c" gracePeriod=30
Oct 31 03:46:58.898510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.898479    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:46:58.898750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:58.898727    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" podUID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" containerName="payload-pre-processing" containerID="cri-o://fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc" gracePeriod=30
Oct 31 03:46:59.233829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.233774    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:46:59.238474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.238448    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:46:59.323121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.323069    5831 generic.go:356] "Generic (PLEG): container finished" podID="288aec47-ab90-45a2-a58e-90bccd98f0cd" containerID="5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c" exitCode=0
Oct 31 03:46:59.323121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.323122    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"
Oct 31 03:46:59.323361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.323140    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" event={"ID":"288aec47-ab90-45a2-a58e-90bccd98f0cd","Type":"ContainerDied","Data":"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"}
Oct 31 03:46:59.323361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.323213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq" event={"ID":"288aec47-ab90-45a2-a58e-90bccd98f0cd","Type":"ContainerDied","Data":"59cd07a21fc6fda3acd8f27125c41550058bba5fb88bebb59fdf5b1bcb08a3e4"}
Oct 31 03:46:59.323361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.323245    5831 scope.go:122] "RemoveContainer" containerID="5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"
Oct 31 03:46:59.324829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.324803    5831 generic.go:356] "Generic (PLEG): container finished" podID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" containerID="fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc" exitCode=0
Oct 31 03:46:59.324977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.324843    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"
Oct 31 03:46:59.324977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.324886    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" event={"ID":"d2e7a4ef-a65a-483b-b5be-ceffbf518e03","Type":"ContainerDied","Data":"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"}
Oct 31 03:46:59.324977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.324935    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv" event={"ID":"d2e7a4ef-a65a-483b-b5be-ceffbf518e03","Type":"ContainerDied","Data":"8bb10f71218e3db4ff332eb58cf22db33960fe10fbf48af2484bbf9b5942f0fb"}
Oct 31 03:46:59.324977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.324951    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"
Oct 31 03:46:59.334064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.334049    5831 scope.go:122] "RemoveContainer" containerID="5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"
Oct 31 03:46:59.334363 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:59.334340    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c\": container with ID starting with 5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c not found: ID does not exist" containerID="5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"
Oct 31 03:46:59.334403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.334379    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c"} err="failed to get container status \"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c\": rpc error: code = NotFound desc = could not find container \"5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c\": container with ID starting with 5ab31709bcd79b6ed771ecb43c3a5e49044f9c4ecf2b2f9974949e8cd022951c not found: ID does not exist"
Oct 31 03:46:59.334449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.334401    5831 scope.go:122] "RemoveContainer" containerID="fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"
Oct 31 03:46:59.347211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.347192    5831 scope.go:122] "RemoveContainer" containerID="fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"
Oct 31 03:46:59.347620 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:46:59.347591    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc\": container with ID starting with fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc not found: ID does not exist" containerID="fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"
Oct 31 03:46:59.347755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.347627    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc"} err="failed to get container status \"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc\": rpc error: code = NotFound desc = could not find container \"fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc\": container with ID starting with fd6e21bc87140bf61e53480ed19129238382c0a30d58f6abcd638b0c459d99fc not found: ID does not exist"
Oct 31 03:46:59.352971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.352947    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\") pod \"288aec47-ab90-45a2-a58e-90bccd98f0cd\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") "
Oct 31 03:46:59.353127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353108    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\") pod \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") "
Oct 31 03:46:59.353289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353263    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\" (UniqueName: \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\") pod \"288aec47-ab90-45a2-a58e-90bccd98f0cd\" (UID: \"288aec47-ab90-45a2-a58e-90bccd98f0cd\") "
Oct 31 03:46:59.353364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353349    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\" (UniqueName: \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\") pod \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\" (UID: \"d2e7a4ef-a65a-483b-b5be-ceffbf518e03\") "
Oct 31 03:46:59.353515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume" pod "d2e7a4ef-a65a-483b-b5be-ceffbf518e03" (UID: "d2e7a4ef-a65a-483b-b5be-ceffbf518e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:59.353515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353494    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume" pod "288aec47-ab90-45a2-a58e-90bccd98f0cd" (UID: "288aec47-ab90-45a2-a58e-90bccd98f0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:59.353948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353910    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288aec47-ab90-45a2-a58e-90bccd98f0cd-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:59.353948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.353937    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:59.356152 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.356115    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp" pod "d2e7a4ef-a65a-483b-b5be-ceffbf518e03" (UID: "d2e7a4ef-a65a-483b-b5be-ceffbf518e03"). InnerVolumeSpecName "kube-api-access-pptvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:59.356322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.356289    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj" pod "288aec47-ab90-45a2-a58e-90bccd98f0cd" (UID: "288aec47-ab90-45a2-a58e-90bccd98f0cd"). InnerVolumeSpecName "kube-api-access-8ffwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:59.471550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.467979    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffwj\" (UniqueName: \"kubernetes.io/projected/288aec47-ab90-45a2-a58e-90bccd98f0cd-kube-api-access-8ffwj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:59.471550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.468011    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptvp\" (UniqueName: \"kubernetes.io/projected/d2e7a4ef-a65a-483b-b5be-ceffbf518e03-kube-api-access-pptvp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:59.483600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.472629    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:46:59.483600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.478503    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d22078-7c749c7b4c-p46xh"]
Oct 31 03:46:59.741786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.741649    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:46:59.746473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.746447    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a32cdb-b45ffb8cf-qgjfv"]
Oct 31 03:46:59.846009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.845971    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:46:59.849227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:46:59.849194    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a32cdb-6695c6bf6d-kwxpq"]
Oct 31 03:47:00.042904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.042819    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:47:00.043715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043695    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043717    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043739    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" containerName="payload-pre-processing"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043748    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" containerName="payload-pre-processing"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043764    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8b68dc18-e915-4c58-9693-42796ac20fa5" containerName="payload-pre-processing"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043772    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b68dc18-e915-4c58-9693-42796ac20fa5" containerName="payload-pre-processing"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043784    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" containerName="revoke-keys"
Oct 31 03:47:00.043797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043791    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" containerName="revoke-keys"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043814    5831 cpu_manager.go:414] "containerMap: removing container" podUID="288aec47-ab90-45a2-a58e-90bccd98f0cd" containerName="payload-processing"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043820    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288aec47-ab90-45a2-a58e-90bccd98f0cd" containerName="payload-processing"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043843    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043849    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.043995    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead5ac1b-c6b4-4ea7-9bc2-8423134fa824" containerName="istio-proxy"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044009    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" containerName="payload-pre-processing"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044019    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b68dc18-e915-4c58-9693-42796ac20fa5" containerName="payload-pre-processing"
Oct 31 03:47:00.044033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044030    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" containerName="revoke-keys"
Oct 31 03:47:00.044285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044043    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="288aec47-ab90-45a2-a58e-90bccd98f0cd" containerName="payload-processing"
Oct 31 03:47:00.044285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044053    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:00.044285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.044063    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:00.049038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.049014    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.097859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.097822    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fc85c7-serving-cert\""
Oct 31 03:47:00.101139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.101107    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:47:00.138477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.138448    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288aec47-ab90-45a2-a58e-90bccd98f0cd" path="/var/lib/kubelet/pods/288aec47-ab90-45a2-a58e-90bccd98f0cd/volumes"
Oct 31 03:47:00.138894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.138877    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b68dc18-e915-4c58-9693-42796ac20fa5" path="/var/lib/kubelet/pods/8b68dc18-e915-4c58-9693-42796ac20fa5/volumes"
Oct 31 03:47:00.139223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.139212    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e7a4ef-a65a-483b-b5be-ceffbf518e03" path="/var/lib/kubelet/pods/d2e7a4ef-a65a-483b-b5be-ceffbf518e03/volumes"
Oct 31 03:47:00.176743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.176712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.176901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.176826    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wg\" (UniqueName: \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.176956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.176940    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.177050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.177037    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.278743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.278708    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.279183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.278800    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.279183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.278820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.279183 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:00.278855    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-serving-cert: secret "maas-api-e2e-praxis-b-fc85c7-serving-cert" not found
Oct 31 03:47:00.279183 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:00.278934    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls podName:2c28d428-0962-4010-b640-3b601f53297e nodeName:}" failed. No retries permitted until 2026-10-31 03:47:00.778913423 +0000 UTC m=+2370.793475127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls") pod "maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" (UID: "2c28d428-0962-4010-b640-3b601f53297e") : secret "maas-api-e2e-praxis-b-fc85c7-serving-cert" not found
Oct 31 03:47:00.279183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.278957    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wg\" (UniqueName: \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.279578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.279558    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.281408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.281390    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.337598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.337485    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wg\" (UniqueName: \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.786575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.786510    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.789101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.789073    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:00.963294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:00.963244    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:01.340873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:01.340842    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:47:01.348297 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:01.347148    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c28d428_0962_4010_b640_3b601f53297e.slice/crio-64bf435a828360343fe4b85bcfc257d21483bf9f596374d4af1ceec59ae34493 WatchSource:0}: Error finding container 64bf435a828360343fe4b85bcfc257d21483bf9f596374d4af1ceec59ae34493: Status 404 returned error can't find the container with id 64bf435a828360343fe4b85bcfc257d21483bf9f596374d4af1ceec59ae34493
Oct 31 03:47:01.349572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:01.349516    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:47:02.363226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:02.363187    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" event={"ID":"2c28d428-0962-4010-b640-3b601f53297e","Type":"ContainerStarted","Data":"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"}
Oct 31 03:47:02.363736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:02.363258    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" event={"ID":"2c28d428-0962-4010-b640-3b601f53297e","Type":"ContainerStarted","Data":"64bf435a828360343fe4b85bcfc257d21483bf9f596374d4af1ceec59ae34493"}
Oct 31 03:47:02.363736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:02.363409    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:02.496872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:02.496816    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" podStartSLOduration=1.581943045 podStartE2EDuration="2.496799826s" totalImagesPullingTime="914.856781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:02.493148552 +0000 UTC m=+2372.507710261" watchObservedRunningTime="2026-10-31 03:47:02.496799826 +0000 UTC m=+2372.511361533"
Oct 31 03:47:05.624861 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:05.624819    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:47:05.625362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:05.625073    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy" containerID="cri-o://aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0" gracePeriod=30
Oct 31 03:47:07.869926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:07.869890    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:47:07.870320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:07.870150    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy" containerID="cri-o://91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c" gracePeriod=30
Oct 31 03:47:08.378291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:08.378261    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:47:10.919188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.919161    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:47:10.961740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961702    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.961940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961759    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.961940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961803    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.961940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961831    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.961940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961855    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.961940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961908    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.962229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.961948    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.962229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962018    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.962229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962058    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:10.962229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962116    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\") pod \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\" (UID: \"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a\") "
Oct 31 03:47:10.962545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962502    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:10.962657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962420    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:10.962657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962598    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:10.962811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962786    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:10.962937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.962895    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:10.964381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.964358    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:10.964902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.964875    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:10.965010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.964895    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "kube-api-access-jdwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:10.965010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:10.964931    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token" pod "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" (UID: "1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:11.064011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.063970    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwl8\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-kube-api-access-jdwl8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064005    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064020    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064034    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064046    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064057    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064071    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.064286 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.064083    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:11.324903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.324868    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:47:11.325614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325591    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:11.325614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325610    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a052139c-cb90-4787-85b2-63bac51e0aed" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325623    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325628    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325639    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325643    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325778    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.325806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.325788    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerName="istio-proxy"
Oct 31 03:47:11.329594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.329575    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.366456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.366419    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26t66\" (UniqueName: \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.366641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.366468    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.378879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.378742    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-14600f\""
Oct 31 03:47:11.378879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.378871    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-14600f-dockercfg-wm6bq\""
Oct 31 03:47:11.383326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.383298    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:47:11.439785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.439754    5831 generic.go:356] "Generic (PLEG): container finished" podID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" containerID="aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0" exitCode=0
Oct 31 03:47:11.439962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.439799    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerDied","Data":"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"}
Oct 31 03:47:11.439962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.439809    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"
Oct 31 03:47:11.439962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.439849    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5" event={"ID":"1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a","Type":"ContainerDied","Data":"2d93bc9458e07677322a447a56a796ddcf93040817df65b4d36ee191b845a7bf"}
Oct 31 03:47:11.439962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.439883    5831 scope.go:122] "RemoveContainer" containerID="aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"
Oct 31 03:47:11.451954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.451930    5831 scope.go:122] "RemoveContainer" containerID="7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"
Oct 31 03:47:11.462676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.462656    5831 scope.go:122] "RemoveContainer" containerID="aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"
Oct 31 03:47:11.462982 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:11.462960    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0\": container with ID starting with aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0 not found: ID does not exist" containerID="aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"
Oct 31 03:47:11.463064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.462995    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0"} err="failed to get container status \"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0\": rpc error: code = NotFound desc = could not find container \"aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0\": container with ID starting with aabbf1cd71b6131724af6d2e694214b5ceb653f70f821d4d9913f26e4c499ce0 not found: ID does not exist"
Oct 31 03:47:11.463064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.463018    5831 scope.go:122] "RemoveContainer" containerID="7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"
Oct 31 03:47:11.463305 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:11.463278    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b\": container with ID starting with 7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b not found: ID does not exist" containerID="7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"
Oct 31 03:47:11.463351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.463313    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b"} err="failed to get container status \"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b\": rpc error: code = NotFound desc = could not find container \"7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b\": container with ID starting with 7bd05ab085d43a35d2ecd969f5857bbe39cb20c46d463190d16124d316ee369b not found: ID does not exist"
Oct 31 03:47:11.468002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.467980    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26t66\" (UniqueName: \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.468090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.468019    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.468696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.468674    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.520732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.520696    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26t66\" (UniqueName: \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\") pod \"payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:11.595681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.595596    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:47:11.599959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.599929    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d22078-openshift-default-5cf4bf8985-f66l5"]
Oct 31 03:47:11.646109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:11.646080    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:12.023613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.023583    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:47:12.025870 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:12.025845    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7287dd64_1dc1_4fbb_ade0_4d2fa7e4331f.slice/crio-e90d6336bb32f971d183ac08cc711791569a054c8cf1d6f10b083efbb54fd2a5 WatchSource:0}: Error finding container e90d6336bb32f971d183ac08cc711791569a054c8cf1d6f10b083efbb54fd2a5: Status 404 returned error can't find the container with id e90d6336bb32f971d183ac08cc711791569a054c8cf1d6f10b083efbb54fd2a5
Oct 31 03:47:12.104424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.104398    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:47:12.110627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.110603    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.148843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.148804    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a" path="/var/lib/kubelet/pods/1f7f4dc9-7ab8-4cdf-aa20-0cd3402fd48a/volumes"
Oct 31 03:47:12.164590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.164561    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:47:12.176649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.176615    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9r7\" (UniqueName: \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.176805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.176788    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.278187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.278095    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.278354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.278230    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9r7\" (UniqueName: \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.278894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.278868    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.331436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.331404    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9r7\" (UniqueName: \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\") pod \"payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.425867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.425827    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:12.451059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.451016    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" event={"ID":"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f","Type":"ContainerStarted","Data":"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"}
Oct 31 03:47:12.451177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.451092    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" event={"ID":"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f","Type":"ContainerStarted","Data":"e90d6336bb32f971d183ac08cc711791569a054c8cf1d6f10b083efbb54fd2a5"}
Oct 31 03:47:12.451177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.451126    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:12.568994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.568890    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" podStartSLOduration=1.5688750470000001 podStartE2EDuration="1.568875047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:12.567109474 +0000 UTC m=+2382.581671185" watchObservedRunningTime="2026-10-31 03:47:12.568875047 +0000 UTC m=+2382.583436755"
Oct 31 03:47:12.827655 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:12.827604    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d0b2ed_2381_49d5_8309_02412b2b3824.slice/crio-970a5e2d9540ec3f8fca9760ca5b30c8bbca39cf47afb6512ad3fb8a185cff4f WatchSource:0}: Error finding container 970a5e2d9540ec3f8fca9760ca5b30c8bbca39cf47afb6512ad3fb8a185cff4f: Status 404 returned error can't find the container with id 970a5e2d9540ec3f8fca9760ca5b30c8bbca39cf47afb6512ad3fb8a185cff4f
Oct 31 03:47:12.828070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:12.828043    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:47:12.979575 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:12.978807    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:47:13.156375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.156347    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:47:13.189970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.189937    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.189999    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190031    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190081    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190345 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190330    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190384    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190444    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190501 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190485    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\") pod \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\" (UID: \"83dddabc-e18b-4d28-8dcc-ff1a774979bd\") "
Oct 31 03:47:13.190993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.190971    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:13.191595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.191523    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:13.193001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.192967    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:13.193404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.193382    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:13.193512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.193441    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:13.193700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.193681    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "kube-api-access-4wlzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:13.193926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.193904    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:13.193991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.193944    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:13.194862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.194845    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy" pod "83dddabc-e18b-4d28-8dcc-ff1a774979bd" (UID: "83dddabc-e18b-4d28-8dcc-ff1a774979bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:13.291703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291659    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291691    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291706    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291719    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291732    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291744    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291755    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291768    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83dddabc-e18b-4d28-8dcc-ff1a774979bd-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.291974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.291782    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wlzm\" (UniqueName: \"kubernetes.io/projected/83dddabc-e18b-4d28-8dcc-ff1a774979bd-kube-api-access-4wlzm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:13.460904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.460862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" event={"ID":"f9d0b2ed-2381-49d5-8309-02412b2b3824","Type":"ContainerStarted","Data":"13b48afe81143e2a1d65942e413fe469222000eeb1245809a57ecde980a79177"}
Oct 31 03:47:13.461105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.460928    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" event={"ID":"f9d0b2ed-2381-49d5-8309-02412b2b3824","Type":"ContainerStarted","Data":"970a5e2d9540ec3f8fca9760ca5b30c8bbca39cf47afb6512ad3fb8a185cff4f"}
Oct 31 03:47:13.461105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.460956    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:13.463014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.462987    5831 generic.go:356] "Generic (PLEG): container finished" podID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerID="91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c" exitCode=0
Oct 31 03:47:13.463159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.463029    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"
Oct 31 03:47:13.463159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.463067    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerDied","Data":"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"}
Oct 31 03:47:13.463159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.463122    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt" event={"ID":"83dddabc-e18b-4d28-8dcc-ff1a774979bd","Type":"ContainerDied","Data":"8bde45454dc75715ee12569da09b86855c88d3250feb23d6b0887cd75bbfcef0"}
Oct 31 03:47:13.463159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.463152    5831 scope.go:122] "RemoveContainer" containerID="91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"
Oct 31 03:47:13.476740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.476716    5831 scope.go:122] "RemoveContainer" containerID="aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"
Oct 31 03:47:13.488285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.488262    5831 scope.go:122] "RemoveContainer" containerID="91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"
Oct 31 03:47:13.488634 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:13.488598    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c\": container with ID starting with 91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c not found: ID does not exist" containerID="91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"
Oct 31 03:47:13.488739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.488645    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c"} err="failed to get container status \"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c\": rpc error: code = NotFound desc = could not find container \"91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c\": container with ID starting with 91bd80b6eab962e559f0977ffc70afaee824d44e493d24dae780e828a230f03c not found: ID does not exist"
Oct 31 03:47:13.488739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.488671    5831 scope.go:122] "RemoveContainer" containerID="aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"
Oct 31 03:47:13.489002 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:13.488968    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7\": container with ID starting with aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7 not found: ID does not exist" containerID="aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"
Oct 31 03:47:13.489103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.489007    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7"} err="failed to get container status \"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7\": rpc error: code = NotFound desc = could not find container \"aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7\": container with ID starting with aa508b419e8b76e08cf352a79a597af7020c8e22e5c045168c334a1a8abfe4b7 not found: ID does not exist"
Oct 31 03:47:13.568098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.568033    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" podStartSLOduration=1.568016209 podStartE2EDuration="1.568016209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:13.565767992 +0000 UTC m=+2383.580329702" watchObservedRunningTime="2026-10-31 03:47:13.568016209 +0000 UTC m=+2383.582577931"
Oct 31 03:47:13.672169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.672140    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:47:13.676733 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:13.676235    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a32cdb-openshift-default-677f4f8d69-p46kt"]
Oct 31 03:47:14.138329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:14.138288    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" path="/var/lib/kubelet/pods/83dddabc-e18b-4d28-8dcc-ff1a774979bd/volumes"
Oct 31 03:47:20.627870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.627835    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"]
Oct 31 03:47:20.628707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.628681    5831 cpu_manager.go:414] "containerMap: removing container" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:20.628707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.628703    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:20.628952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.628844    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:20.628952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.628861    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:20.633785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.633764    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.681999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.681965    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.682378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.682329    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.682508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.682472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsph\" (UniqueName: \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.683817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.683794    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"]
Oct 31 03:47:20.725081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.725036    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:20.725081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.725067    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:47:20.783651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.783613    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsph\" (UniqueName: \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.783842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.783813    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.783900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.783880    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.784716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.784690    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.786479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.786453    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.836213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.836182    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsph\" (UniqueName: \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\") pod \"maas-api-revoke-sub-b2d4b1a6-6s5wx\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:20.951186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:20.951149    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:21.334311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:21.334281    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"]
Oct 31 03:47:21.336588 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:21.336557    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de38b6d_be48_47d1_9a11_d3f1b2c817bc.slice/crio-1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0 WatchSource:0}: Error finding container 1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0: Status 404 returned error can't find the container with id 1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0
Oct 31 03:47:21.543289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:21.543257    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx" event={"ID":"5de38b6d-be48-47d1-9a11-d3f1b2c817bc","Type":"ContainerStarted","Data":"1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0"}
Oct 31 03:47:22.551792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:22.551708    5831 generic.go:356] "Generic (PLEG): container finished" podID="5de38b6d-be48-47d1-9a11-d3f1b2c817bc" containerID="4d8c627d05a28cbd5477397b9781c7d3285293efc8a884175c2d49e3a84d2a4f" exitCode=0
Oct 31 03:47:22.551792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:22.551745    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx" event={"ID":"5de38b6d-be48-47d1-9a11-d3f1b2c817bc","Type":"ContainerDied","Data":"4d8c627d05a28cbd5477397b9781c7d3285293efc8a884175c2d49e3a84d2a4f"}
Oct 31 03:47:22.692812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:22.692773    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:47:23.486574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.486518    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:47:23.771675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.771618    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:23.831591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.831519    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\" (UniqueName: \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\") pod \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") "
Oct 31 03:47:23.831775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.831699    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\") pod \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") "
Oct 31 03:47:23.831775 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.831729    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\") pod \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\" (UID: \"5de38b6d-be48-47d1-9a11-d3f1b2c817bc\") "
Oct 31 03:47:23.832145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.832115    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca" pod "5de38b6d-be48-47d1-9a11-d3f1b2c817bc" (UID: "5de38b6d-be48-47d1-9a11-d3f1b2c817bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:23.834114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.834087    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca" pod "5de38b6d-be48-47d1-9a11-d3f1b2c817bc" (UID: "5de38b6d-be48-47d1-9a11-d3f1b2c817bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:23.834336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.834312    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph" pod "5de38b6d-be48-47d1-9a11-d3f1b2c817bc" (UID: "5de38b6d-be48-47d1-9a11-d3f1b2c817bc"). InnerVolumeSpecName "kube-api-access-8lsph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:23.933186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.933148    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsph\" (UniqueName: \"kubernetes.io/projected/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-kube-api-access-8lsph\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:23.933186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.933179    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:23.933186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:23.933189    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de38b6d-be48-47d1-9a11-d3f1b2c817bc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.473698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:24.473664    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:47:24.568016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:24.567985    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"
Oct 31 03:47:24.568199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:24.568021    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx" event={"ID":"5de38b6d-be48-47d1-9a11-d3f1b2c817bc","Type":"ContainerDied","Data":"1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0"}
Oct 31 03:47:24.568199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:24.568066    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9cfc8800d75690b329da8fdd1383392f3f0b958aedccf30623c2642a62a0b0"
Oct 31 03:47:26.085067 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085031    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:47:26.085663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085648    5831 cpu_manager.go:414] "containerMap: removing container" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:26.085747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085664    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83dddabc-e18b-4d28-8dcc-ff1a774979bd" containerName="istio-proxy"
Oct 31 03:47:26.085747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085694    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5de38b6d-be48-47d1-9a11-d3f1b2c817bc" containerName="revoke-keys"
Oct 31 03:47:26.085747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085699    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de38b6d-be48-47d1-9a11-d3f1b2c817bc" containerName="revoke-keys"
Oct 31 03:47:26.085842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.085813    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de38b6d-be48-47d1-9a11-d3f1b2c817bc" containerName="revoke-keys"
Oct 31 03:47:26.089437 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.089422    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.137351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.137312    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ca15df-openshift-default-dockercfg-x4vxf\""
Oct 31 03:47:26.159803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.159771    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.159985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.159814    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.159985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.159911    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rv9\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160073    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160109    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160125    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160165    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160241    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.160344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.160328    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.177328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.177296    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:47:26.262322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.262279    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.277671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.277631    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.277956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.277936    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rv9\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.278224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.278208    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.278340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.278327    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.278449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.278434    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.278631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.278612    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.293606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.293567    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.293892 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.293875    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.294588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.294565    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.295046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.295023    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.307451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.307424    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.322153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.308449    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.322153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.308741    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.322153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.309419    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.322945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.322911    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.332707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.332674    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.349369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.349287    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rv9\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\") pod \"e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") " pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.405121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.405088    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:26.827939 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.827911    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:47:26.830037 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:26.830002    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbd4482_dc64_44e8_869a_be9f06d41363.slice/crio-61aaa7a3f92034b72e151f703e7d8aeb5bfc53543e7457b18fae194b24d43891 WatchSource:0}: Error finding container 61aaa7a3f92034b72e151f703e7d8aeb5bfc53543e7457b18fae194b24d43891: Status 404 returned error can't find the container with id 61aaa7a3f92034b72e151f703e7d8aeb5bfc53543e7457b18fae194b24d43891
Oct 31 03:47:26.832926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.832893    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:26.833022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.832974    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:26.833022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:26.833014    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:27.571895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.571862    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:47:27.572381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.572083    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" podUID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" containerName="maas-api" containerID="cri-o://c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b" gracePeriod=30
Oct 31 03:47:27.598847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.598817    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerStarted","Data":"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"}
Oct 31 03:47:27.598954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.598889    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerStarted","Data":"61aaa7a3f92034b72e151f703e7d8aeb5bfc53543e7457b18fae194b24d43891"}
Oct 31 03:47:27.786572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.786471    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podStartSLOduration=1.786453634 podStartE2EDuration="1.786453634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:27.785789361 +0000 UTC m=+2397.800351065" watchObservedRunningTime="2026-10-31 03:47:27.786453634 +0000 UTC m=+2397.801015342"
Oct 31 03:47:27.894945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:27.894918    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:47:28.031108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.030898    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\") pod \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") "
Oct 31 03:47:28.031108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.030979    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\") pod \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") "
Oct 31 03:47:28.031108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.031046    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\" (UniqueName: \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\") pod \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") "
Oct 31 03:47:28.031452 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.031136    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") pod \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\" (UID: \"7909c7e4-b1a4-4476-a2f0-78bab221b44d\") "
Oct 31 03:47:28.055934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.055867    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca" pod "7909c7e4-b1a4-4476-a2f0-78bab221b44d" (UID: "7909c7e4-b1a4-4476-a2f0-78bab221b44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:28.066189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.066152    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls" pod "7909c7e4-b1a4-4476-a2f0-78bab221b44d" (UID: "7909c7e4-b1a4-4476-a2f0-78bab221b44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:28.066379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.066308    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt" pod "7909c7e4-b1a4-4476-a2f0-78bab221b44d" (UID: "7909c7e4-b1a4-4476-a2f0-78bab221b44d"). InnerVolumeSpecName "kube-api-access-zvvtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:28.066451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.066403    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs" pod "7909c7e4-b1a4-4476-a2f0-78bab221b44d" (UID: "7909c7e4-b1a4-4476-a2f0-78bab221b44d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:28.067235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.067207    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:47:28.067493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.067455    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" podUID="a325465a-6593-4466-b0a4-e260b56e298e" containerName="payload-processing" containerID="cri-o://a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7" gracePeriod=30
Oct 31 03:47:28.132906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.132616    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.132906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.132862    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7909c7e4-b1a4-4476-a2f0-78bab221b44d-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.132906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.132884    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7909c7e4-b1a4-4476-a2f0-78bab221b44d-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.132906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.132899    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvtt\" (UniqueName: \"kubernetes.io/projected/7909c7e4-b1a4-4476-a2f0-78bab221b44d-kube-api-access-zvvtt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.340683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.340659    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:47:28.405652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.405572    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:28.411513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.411489    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:28.435964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.435935    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\") pod \"a325465a-6593-4466-b0a4-e260b56e298e\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") "
Oct 31 03:47:28.436127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.436113    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\" (UniqueName: \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\") pod \"a325465a-6593-4466-b0a4-e260b56e298e\" (UID: \"a325465a-6593-4466-b0a4-e260b56e298e\") "
Oct 31 03:47:28.436331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.436303    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume" pod "a325465a-6593-4466-b0a4-e260b56e298e" (UID: "a325465a-6593-4466-b0a4-e260b56e298e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:28.436583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.436566    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325465a-6593-4466-b0a4-e260b56e298e-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.438454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.438431    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv" pod "a325465a-6593-4466-b0a4-e260b56e298e" (UID: "a325465a-6593-4466-b0a4-e260b56e298e"). InnerVolumeSpecName "kube-api-access-r26sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:28.537262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.537229    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26sv\" (UniqueName: \"kubernetes.io/projected/a325465a-6593-4466-b0a4-e260b56e298e-kube-api-access-r26sv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.635557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.634851    5831 generic.go:356] "Generic (PLEG): container finished" podID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" containerID="c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b" exitCode=0
Oct 31 03:47:28.635557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.635053    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" event={"ID":"7909c7e4-b1a4-4476-a2f0-78bab221b44d","Type":"ContainerDied","Data":"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"}
Oct 31 03:47:28.635557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.635103    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7" event={"ID":"7909c7e4-b1a4-4476-a2f0-78bab221b44d","Type":"ContainerDied","Data":"e543d5aef92a60d86b2881acab5167ddafe44e64a3a9ed9ff84885b0755dbff2"}
Oct 31 03:47:28.635557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.635134    5831 scope.go:122] "RemoveContainer" containerID="c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"
Oct 31 03:47:28.635557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.635306    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"
Oct 31 03:47:28.646185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.646154    5831 generic.go:356] "Generic (PLEG): container finished" podID="a325465a-6593-4466-b0a4-e260b56e298e" containerID="a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7" exitCode=0
Oct 31 03:47:28.646438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.646231    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" event={"ID":"a325465a-6593-4466-b0a4-e260b56e298e","Type":"ContainerDied","Data":"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"}
Oct 31 03:47:28.646583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.646492    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb" event={"ID":"a325465a-6593-4466-b0a4-e260b56e298e","Type":"ContainerDied","Data":"8ba6f0a1029a337c2e6978682b2cb619aefd5d72aa581ff3a346ed7c700f4d89"}
Oct 31 03:47:28.646583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.646267    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"
Oct 31 03:47:28.647153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.647130    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:28.648040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.647978    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:47:28.657468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.657451    5831 scope.go:122] "RemoveContainer" containerID="c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"
Oct 31 03:47:28.657825 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:28.657802    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b\": container with ID starting with c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b not found: ID does not exist" containerID="c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"
Oct 31 03:47:28.657927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.657836    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b"} err="failed to get container status \"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b\": rpc error: code = NotFound desc = could not find container \"c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b\": container with ID starting with c5be88de7a429bb864f7963eb762ff8b5f010b06145740f351cfad834c1ff61b not found: ID does not exist"
Oct 31 03:47:28.657927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.657858    5831 scope.go:122] "RemoveContainer" containerID="a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"
Oct 31 03:47:28.667679 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.667662    5831 scope.go:122] "RemoveContainer" containerID="a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"
Oct 31 03:47:28.667977 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:28.667947    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7\": container with ID starting with a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7 not found: ID does not exist" containerID="a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"
Oct 31 03:47:28.668059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.667986    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7"} err="failed to get container status \"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7\": rpc error: code = NotFound desc = could not find container \"a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7\": container with ID starting with a9559652d9d51d7cfb8ac02e5b022850217a7b569b50114800b66775d8b960c7 not found: ID does not exist"
Oct 31 03:47:28.903421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.903385    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:47:28.906766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:28.906698    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c561e3-c7bcdfc7d-gs7pb"]
Oct 31 03:47:29.005642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:29.005592    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:47:29.008623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:29.008596    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c561e3-67fc48d6d8-lwsx7"]
Oct 31 03:47:30.138761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:30.138730    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" path="/var/lib/kubelet/pods/7909c7e4-b1a4-4476-a2f0-78bab221b44d/volumes"
Oct 31 03:47:30.139226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:30.139185    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a325465a-6593-4466-b0a4-e260b56e298e" path="/var/lib/kubelet/pods/a325465a-6593-4466-b0a4-e260b56e298e/volumes"
Oct 31 03:47:34.410320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.410286    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:47:34.411163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411143    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" containerName="maas-api"
Oct 31 03:47:34.411223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411167    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" containerName="maas-api"
Oct 31 03:47:34.411223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411185    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a325465a-6593-4466-b0a4-e260b56e298e" containerName="payload-processing"
Oct 31 03:47:34.411223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411193    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a325465a-6593-4466-b0a4-e260b56e298e" containerName="payload-processing"
Oct 31 03:47:34.411413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411399    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7909c7e4-b1a4-4476-a2f0-78bab221b44d" containerName="maas-api"
Oct 31 03:47:34.411448 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.411419    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a325465a-6593-4466-b0a4-e260b56e298e" containerName="payload-processing"
Oct 31 03:47:34.417327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.417307    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.428150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.428112    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkj5\" (UniqueName: \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.428294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.428219    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.469877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.469835    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:47:34.508036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.507883    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca15df\""
Oct 31 03:47:34.508036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.507941    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca15df-dockercfg-9f2vz\""
Oct 31 03:47:34.564878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.553602    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.564878 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.564518    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkj5\" (UniqueName: \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.603396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.603365    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.621364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.621328    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:47:34.625184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.625095    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkj5\" (UniqueName: \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.626673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.626653    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.640310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.640003    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:47:34.665755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.665668    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x8v\" (UniqueName: \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.665925 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.665838    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.734901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.734872    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:34.767578 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.767509    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.767746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.767729    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x8v\" (UniqueName: \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.768189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.768171    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.821303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.821273    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x8v\" (UniqueName: \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\") pod \"payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:34.950983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:34.950955    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:35.106562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.106518    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:47:35.107708 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:35.107684    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c506e7_8b43_4bcf_8c2d_2fdc1079a6b7.slice/crio-232c6798e9277dee14da7319b9a190bb1034fc2d90c8507d82107e6d088eeae9 WatchSource:0}: Error finding container 232c6798e9277dee14da7319b9a190bb1034fc2d90c8507d82107e6d088eeae9: Status 404 returned error can't find the container with id 232c6798e9277dee14da7319b9a190bb1034fc2d90c8507d82107e6d088eeae9
Oct 31 03:47:35.322385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.322350    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:47:35.324607 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:35.324573    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a26d0ee_10bf_4dd1_9580_4bbce4ace5c1.slice/crio-3c3697e2b1926d1880b01a961891195a9d8d98a8a13501bf5122ee9419bd9747 WatchSource:0}: Error finding container 3c3697e2b1926d1880b01a961891195a9d8d98a8a13501bf5122ee9419bd9747: Status 404 returned error can't find the container with id 3c3697e2b1926d1880b01a961891195a9d8d98a8a13501bf5122ee9419bd9747
Oct 31 03:47:35.705688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.705655    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" event={"ID":"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7","Type":"ContainerStarted","Data":"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"}
Oct 31 03:47:35.706148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.705717    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" event={"ID":"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7","Type":"ContainerStarted","Data":"232c6798e9277dee14da7319b9a190bb1034fc2d90c8507d82107e6d088eeae9"}
Oct 31 03:47:35.706148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.705754    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:35.707433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.707404    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" event={"ID":"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1","Type":"ContainerStarted","Data":"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"}
Oct 31 03:47:35.707584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.707453    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" event={"ID":"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1","Type":"ContainerStarted","Data":"3c3697e2b1926d1880b01a961891195a9d8d98a8a13501bf5122ee9419bd9747"}
Oct 31 03:47:35.707584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.707481    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:35.810938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.810890    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" podStartSLOduration=1.8108763209999998 podStartE2EDuration="1.810876321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:35.810381727 +0000 UTC m=+2405.824943434" watchObservedRunningTime="2026-10-31 03:47:35.810876321 +0000 UTC m=+2405.825438029"
Oct 31 03:47:35.916574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:35.915367    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" podStartSLOduration=1.915347643 podStartE2EDuration="1.915347643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:35.914616623 +0000 UTC m=+2405.929178331" watchObservedRunningTime="2026-10-31 03:47:35.915347643 +0000 UTC m=+2405.929909352"
Oct 31 03:47:38.235503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.235476    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:47:38.236016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.235796    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:47:38.250951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.250923    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:47:38.253669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.253648    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:47:38.255739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.255719    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:47:38.257963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:38.257943    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:47:40.354074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:40.354034    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:47:40.354589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:40.354290    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy" containerID="cri-o://e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5" gracePeriod=30
Oct 31 03:47:42.024556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.016784    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"]
Oct 31 03:47:42.028848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.028823    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.065677 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.065634    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"]
Oct 31 03:47:42.077599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.077569    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:42.107453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.107423    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:47:42.164108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.164075    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvk5k\" (UniqueName: \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.164292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.164124    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.164292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.164208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.265825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.265778    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvk5k\" (UniqueName: \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.265825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.265834    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.266114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.265866    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.266711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.266680    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.268443 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.268422    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.320103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.320020    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvk5k\" (UniqueName: \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\") pod \"maas-api-revoke-sub-a9af1989-bxk9d\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.346905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.346874    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:42.529957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.529915    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:47:42.535562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.535513    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.624000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.606560    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:47:42.624000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.607147    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-de2877-openshift-default-dockercfg-5rbdx\""
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.718762    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.718840    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.718910    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647fq\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.718974    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.718999    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.719047    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.719101    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.719177    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.720561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.719213    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.755673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.755625    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"]
Oct 31 03:47:42.757215 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:42.757187    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5800a46c_8d36_4f52_91d1_ca820856e968.slice/crio-d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018 WatchSource:0}: Error finding container d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018: Status 404 returned error can't find the container with id d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018
Oct 31 03:47:42.788020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.787988    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d" event={"ID":"5800a46c-8d36-4f52-91d1-ca820856e968","Type":"ContainerStarted","Data":"d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018"}
Oct 31 03:47:42.820049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820007    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820068    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820114    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820162    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820213    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647fq\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820268    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820302    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820371    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820405    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820766 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820748    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820767    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820862    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.820900 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.820880    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.822902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.822880    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.823011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.822994    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.877682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.877590    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.877870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.877733    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647fq\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\") pod \"e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") " pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:42.931273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:42.931234    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:43.006762 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:43.006721    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:47:43.398042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.398014    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:47:43.399278 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:43.399252    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00cac98_d2a5_4707_ad96_53bc721e6e4f.slice/crio-714c79b5da0bfe5965ff61e0aec068344c4f78b06d24f61f70c037dc535a4b5b WatchSource:0}: Error finding container 714c79b5da0bfe5965ff61e0aec068344c4f78b06d24f61f70c037dc535a4b5b: Status 404 returned error can't find the container with id 714c79b5da0bfe5965ff61e0aec068344c4f78b06d24f61f70c037dc535a4b5b
Oct 31 03:47:43.402059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.402024    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:43.402161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.402103    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:43.402161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.402135    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:47:43.797239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.797199    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d" event={"ID":"5800a46c-8d36-4f52-91d1-ca820856e968","Type":"ContainerStarted","Data":"f361df647ca83dab691a46dbd55168f56918d78a50dea93fa58d4fdff8f05d88"}
Oct 31 03:47:43.799081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.799050    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerStarted","Data":"9601d6dd5267e5ccf56596a978437563d1a4f3ad66cf5763d03560dcfadbc5a8"}
Oct 31 03:47:43.799229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.799104    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerStarted","Data":"714c79b5da0bfe5965ff61e0aec068344c4f78b06d24f61f70c037dc535a4b5b"}
Oct 31 03:47:43.904863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.904800    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d" podStartSLOduration=2.904780413 podStartE2EDuration="2.904780413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:43.902739017 +0000 UTC m=+2413.917300725" watchObservedRunningTime="2026-10-31 03:47:43.904780413 +0000 UTC m=+2413.919342122"
Oct 31 03:47:43.931912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:43.931883    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:44.011937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:44.011837    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podStartSLOduration=2.011822013 podStartE2EDuration="2.011822013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:44.009105092 +0000 UTC m=+2414.023666794" watchObservedRunningTime="2026-10-31 03:47:44.011822013 +0000 UTC m=+2414.026383722"
Oct 31 03:47:44.807891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:44.807856    5831 generic.go:356] "Generic (PLEG): container finished" podID="5800a46c-8d36-4f52-91d1-ca820856e968" containerID="f361df647ca83dab691a46dbd55168f56918d78a50dea93fa58d4fdff8f05d88" exitCode=0
Oct 31 03:47:44.808303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:44.807941    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d" event={"ID":"5800a46c-8d36-4f52-91d1-ca820856e968","Type":"ContainerDied","Data":"f361df647ca83dab691a46dbd55168f56918d78a50dea93fa58d4fdff8f05d88"}
Oct 31 03:47:44.938029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:44.937995    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:45.649910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.649883    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:47:45.653935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.653913    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.653979    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654017    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654056    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654081    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654103    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654120    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654149    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:45.654379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654267    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654305    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:45.654379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654319    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:45.654379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654366    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\") pod \"9627b327-9b81-4c03-87c2-f5f060fc0609\" (UID: \"9627b327-9b81-4c03-87c2-f5f060fc0609\") "
Oct 31 03:47:45.654619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654408    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:45.654619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.654518    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:45.655133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.655005    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.655133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.655081    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.655133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.655096    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.655133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.655109    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.655133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.655125    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9627b327-9b81-4c03-87c2-f5f060fc0609-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.656577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.656507    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:45.656577 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.656565    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:45.656720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.656600    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "kube-api-access-jrtnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:45.656720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.656714    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy" pod "9627b327-9b81-4c03-87c2-f5f060fc0609" (UID: "9627b327-9b81-4c03-87c2-f5f060fc0609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:45.756866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.756833    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.756866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.756861    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrtnn\" (UniqueName: \"kubernetes.io/projected/9627b327-9b81-4c03-87c2-f5f060fc0609-kube-api-access-jrtnn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.756866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.756874    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.757114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.756883    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9627b327-9b81-4c03-87c2-f5f060fc0609-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:45.820620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.820574    5831 generic.go:356] "Generic (PLEG): container finished" podID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerID="e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5" exitCode=0
Oct 31 03:47:45.820620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.820610    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"
Oct 31 03:47:45.821180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.820645    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerDied","Data":"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"}
Oct 31 03:47:45.821180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.820699    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7" event={"ID":"9627b327-9b81-4c03-87c2-f5f060fc0609","Type":"ContainerDied","Data":"d5ed7b48e76567fc4c9b7ff14f9ea10b89eb4d2232edd99f28c43bb316746464"}
Oct 31 03:47:45.821180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.820729    5831 scope.go:122] "RemoveContainer" containerID="e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"
Oct 31 03:47:45.822401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.821778    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:45.822786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.822727    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:47:45.836480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.836456    5831 scope.go:122] "RemoveContainer" containerID="fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"
Oct 31 03:47:45.849462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.849441    5831 scope.go:122] "RemoveContainer" containerID="e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"
Oct 31 03:47:45.849832 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:45.849802    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5\": container with ID starting with e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5 not found: ID does not exist" containerID="e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"
Oct 31 03:47:45.849919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.849841    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5"} err="failed to get container status \"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5\": rpc error: code = NotFound desc = could not find container \"e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5\": container with ID starting with e2ffc6f565ec5962ffc932132d0f8180b640bb0c430a7d974ec55c7fd941f2f5 not found: ID does not exist"
Oct 31 03:47:45.849919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.849857    5831 scope.go:122] "RemoveContainer" containerID="fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"
Oct 31 03:47:45.850155 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:47:45.850133    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c\": container with ID starting with fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c not found: ID does not exist" containerID="fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"
Oct 31 03:47:45.850213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.850166    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c"} err="failed to get container status \"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c\": rpc error: code = NotFound desc = could not find container \"fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c\": container with ID starting with fe30d3a828d3a768552945b66cd385835270347bce0d605879b98167b92a8b7c not found: ID does not exist"
Oct 31 03:47:45.970503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.970475    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:45.974811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.974783    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:47:45.978817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:45.978794    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c561e3-openshift-default-6c69b648c9-mgkr7"]
Oct 31 03:47:46.060469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.060439    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\" (UniqueName: \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\") pod \"5800a46c-8d36-4f52-91d1-ca820856e968\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") "
Oct 31 03:47:46.060647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.060549    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\") pod \"5800a46c-8d36-4f52-91d1-ca820856e968\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") "
Oct 31 03:47:46.060714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.060651    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\") pod \"5800a46c-8d36-4f52-91d1-ca820856e968\" (UID: \"5800a46c-8d36-4f52-91d1-ca820856e968\") "
Oct 31 03:47:46.060919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.060893    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca" pod "5800a46c-8d36-4f52-91d1-ca820856e968" (UID: "5800a46c-8d36-4f52-91d1-ca820856e968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:46.062865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.062842    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca" pod "5800a46c-8d36-4f52-91d1-ca820856e968" (UID: "5800a46c-8d36-4f52-91d1-ca820856e968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:46.062972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.062904    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k" pod "5800a46c-8d36-4f52-91d1-ca820856e968" (UID: "5800a46c-8d36-4f52-91d1-ca820856e968"). InnerVolumeSpecName "kube-api-access-gvk5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:46.145213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.145123    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" path="/var/lib/kubelet/pods/9627b327-9b81-4c03-87c2-f5f060fc0609/volumes"
Oct 31 03:47:46.162271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.162240    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:46.162440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.162274    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5800a46c-8d36-4f52-91d1-ca820856e968-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:46.162440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.162291    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvk5k\" (UniqueName: \"kubernetes.io/projected/5800a46c-8d36-4f52-91d1-ca820856e968-kube-api-access-gvk5k\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:46.719509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.719476    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:47:46.719761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.719575    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:47:46.834080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.834049    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"
Oct 31 03:47:46.834567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.834089    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d" event={"ID":"5800a46c-8d36-4f52-91d1-ca820856e968","Type":"ContainerDied","Data":"d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018"}
Oct 31 03:47:46.834567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:46.834146    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e6a2feaa7d8b8e9b7a3e99386a63f35ec5572d69458bb3f3b54fd13cb7a018"
Oct 31 03:47:51.113456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.113416    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"]
Oct 31 03:47:51.114332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114313    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5800a46c-8d36-4f52-91d1-ca820856e968" containerName="revoke-keys"
Oct 31 03:47:51.114385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114336    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5800a46c-8d36-4f52-91d1-ca820856e968" containerName="revoke-keys"
Oct 31 03:47:51.114385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114363    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:51.114385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114371    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:51.114486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114429    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:51.114486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114437    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:51.114605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114593    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5800a46c-8d36-4f52-91d1-ca820856e968" containerName="revoke-keys"
Oct 31 03:47:51.114643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.114611    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:51.119821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.119799    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.169411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.169375    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"]
Oct 31 03:47:51.210687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.210644    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:51.210901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.210739    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:47:51.221412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.221387    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkn6p\" (UniqueName: \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.221546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.221429    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.221656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.221634    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.322681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.322644    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.322879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.322755    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.322879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.322864    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkn6p\" (UniqueName: \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.323708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.323504    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.325469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.325439    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.376339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.376270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkn6p\" (UniqueName: \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\") pod \"maas-api-revoke-sub-d58d3705-wr4wd\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.439722 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.439692    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:51.813200 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.813174    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"]
Oct 31 03:47:51.815111 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:51.815081    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9997e0_e6c5_4e59_8fb4_60b576ce9a17.slice/crio-f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6 WatchSource:0}: Error finding container f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6: Status 404 returned error can't find the container with id f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6
Oct 31 03:47:51.869940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:51.869911    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd" event={"ID":"be9997e0-e6c5-4e59-8fb4-60b576ce9a17","Type":"ContainerStarted","Data":"f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6"}
Oct 31 03:47:52.879359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:52.879322    5831 generic.go:356] "Generic (PLEG): container finished" podID="be9997e0-e6c5-4e59-8fb4-60b576ce9a17" containerID="85ab4c65670b7b6e47204bf28f2d527c9eaa60faf494509be558ddcff8610d66" exitCode=0
Oct 31 03:47:52.879805 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:52.879387    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd" event={"ID":"be9997e0-e6c5-4e59-8fb4-60b576ce9a17","Type":"ContainerDied","Data":"85ab4c65670b7b6e47204bf28f2d527c9eaa60faf494509be558ddcff8610d66"}
Oct 31 03:47:53.222265 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.222230    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:47:53.223081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.223056    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9627b327-9b81-4c03-87c2-f5f060fc0609" containerName="istio-proxy"
Oct 31 03:47:53.226714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.226696    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.271125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.270323    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.271125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.270472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j76v\" (UniqueName: \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.305396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.305315    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:47:53.305850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.305825    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-de2877-dockercfg-rtv4j\""
Oct 31 03:47:53.306170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.306110    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-de2877\""
Oct 31 03:47:53.385554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.377038    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.385554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.377239    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j76v\" (UniqueName: \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.385554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.381106    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.390389 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.386156    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:47:53.402334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.393265    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.431510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.431473    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j76v\" (UniqueName: \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\") pod \"payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.446774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.446744    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:47:53.478911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.478817    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhf5\" (UniqueName: \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.479081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.479004    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.541515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.541479    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:53.580367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.580331    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhf5\" (UniqueName: \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.580590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.580451    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.581171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.581143    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.634472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.634437    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhf5\" (UniqueName: \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\") pod \"payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.720372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.720334    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:53.912617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:53.912583    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:47:53.916808 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:53.916774    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c74d1e_340f_4300_86df_ee34b878c46d.slice/crio-8c06757ef67f5966a4acd7faf26ae513963d3ef1df225d3e810bf3c712c10114 WatchSource:0}: Error finding container 8c06757ef67f5966a4acd7faf26ae513963d3ef1df225d3e810bf3c712c10114: Status 404 returned error can't find the container with id 8c06757ef67f5966a4acd7faf26ae513963d3ef1df225d3e810bf3c712c10114
Oct 31 03:47:54.031742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.031718    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:54.088399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.088368    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\" (UniqueName: \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\") pod \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") "
Oct 31 03:47:54.088615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.088504    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\") pod \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") "
Oct 31 03:47:54.088705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.088687    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\") pod \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\" (UID: \"be9997e0-e6c5-4e59-8fb4-60b576ce9a17\") "
Oct 31 03:47:54.089081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.089058    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca" pod "be9997e0-e6c5-4e59-8fb4-60b576ce9a17" (UID: "be9997e0-e6c5-4e59-8fb4-60b576ce9a17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:54.089205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.089184    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:54.091243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.091191    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p" pod "be9997e0-e6c5-4e59-8fb4-60b576ce9a17" (UID: "be9997e0-e6c5-4e59-8fb4-60b576ce9a17"). InnerVolumeSpecName "kube-api-access-xkn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:54.091707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.091292    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca" pod "be9997e0-e6c5-4e59-8fb4-60b576ce9a17" (UID: "be9997e0-e6c5-4e59-8fb4-60b576ce9a17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:54.099687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.099662    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:47:54.102873 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:54.102849    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac691f35_6d32_4809_ab1e_ebc664b0a7d0.slice/crio-e8e2b63cea7979ffeda69efdc14ff2f04f5ea7c6b9dbb94e7838c6b9a3b60f89 WatchSource:0}: Error finding container e8e2b63cea7979ffeda69efdc14ff2f04f5ea7c6b9dbb94e7838c6b9a3b60f89: Status 404 returned error can't find the container with id e8e2b63cea7979ffeda69efdc14ff2f04f5ea7c6b9dbb94e7838c6b9a3b60f89
Oct 31 03:47:54.191020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.190844    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkn6p\" (UniqueName: \"kubernetes.io/projected/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-kube-api-access-xkn6p\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:54.191020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.190880    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9997e0-e6c5-4e59-8fb4-60b576ce9a17-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:54.907412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.907299    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" event={"ID":"67c74d1e-340f-4300-86df-ee34b878c46d","Type":"ContainerStarted","Data":"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"}
Oct 31 03:47:54.907412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.907406    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" event={"ID":"67c74d1e-340f-4300-86df-ee34b878c46d","Type":"ContainerStarted","Data":"8c06757ef67f5966a4acd7faf26ae513963d3ef1df225d3e810bf3c712c10114"}
Oct 31 03:47:54.907715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.907566    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:47:54.908909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.908886    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"
Oct 31 03:47:54.909025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.908927    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd" event={"ID":"be9997e0-e6c5-4e59-8fb4-60b576ce9a17","Type":"ContainerDied","Data":"f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6"}
Oct 31 03:47:54.909025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.908970    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3de8f1dea32628fbcc0275395c53b6a7fe13b9b9ce0a1b6df88041e9a9eb4d6"
Oct 31 03:47:54.910872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.910846    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" event={"ID":"ac691f35-6d32-4809-ab1e-ebc664b0a7d0","Type":"ContainerStarted","Data":"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"}
Oct 31 03:47:54.910996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.910895    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" event={"ID":"ac691f35-6d32-4809-ab1e-ebc664b0a7d0","Type":"ContainerStarted","Data":"e8e2b63cea7979ffeda69efdc14ff2f04f5ea7c6b9dbb94e7838c6b9a3b60f89"}
Oct 31 03:47:54.910996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:54.910920    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:47:55.013953 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:55.013894    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" podStartSLOduration=2.013877842 podStartE2EDuration="2.013877842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:55.012296309 +0000 UTC m=+2425.026858016" watchObservedRunningTime="2026-10-31 03:47:55.013877842 +0000 UTC m=+2425.028439553"
Oct 31 03:47:55.121130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:55.121078    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" podStartSLOduration=2.121064888 podStartE2EDuration="2.121064888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:55.118423698 +0000 UTC m=+2425.132985428" watchObservedRunningTime="2026-10-31 03:47:55.121064888 +0000 UTC m=+2425.135626635"
Oct 31 03:47:58.707629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.707515    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"]
Oct 31 03:47:58.708275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.708254    5831 cpu_manager.go:414] "containerMap: removing container" podUID="be9997e0-e6c5-4e59-8fb4-60b576ce9a17" containerName="revoke-keys"
Oct 31 03:47:58.708347 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.708277    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9997e0-e6c5-4e59-8fb4-60b576ce9a17" containerName="revoke-keys"
Oct 31 03:47:58.708384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.708375    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9997e0-e6c5-4e59-8fb4-60b576ce9a17" containerName="revoke-keys"
Oct 31 03:47:58.714975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.714939    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.753421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.753381    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"]
Oct 31 03:47:58.760203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.760171    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9q67\" (UniqueName: \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.760341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.760223    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.760423 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.760340    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.799569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.795862    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:47:58.799569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.795945    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:58.861479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.861449    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9q67\" (UniqueName: \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.861663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.861491    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.861663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.861590    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.862357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.862335    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.864392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.864368    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:58.915248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:58.915211    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9q67\" (UniqueName: \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\") pod \"maas-api-revoke-sub-00d9738d-kdhj8\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:59.030771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:59.030685    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:47:59.437064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:59.437036    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"]
Oct 31 03:47:59.438953 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:47:59.438929    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c80aded_7843_40c3_b4cc_016fdb0b27fb.slice/crio-f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b WatchSource:0}: Error finding container f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b: Status 404 returned error can't find the container with id f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b
Oct 31 03:47:59.951144 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:59.951114    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8" event={"ID":"4c80aded-7843-40c3-b4cc-016fdb0b27fb","Type":"ContainerStarted","Data":"484d75138df3be136c7de7c859a3aca0b2f6f1ed9c0a7f70b38eb831b25d77de"}
Oct 31 03:47:59.951582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:47:59.951170    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8" event={"ID":"4c80aded-7843-40c3-b4cc-016fdb0b27fb","Type":"ContainerStarted","Data":"f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b"}
Oct 31 03:48:00.060390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:00.060321    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8" podStartSLOduration=2.060301297 podStartE2EDuration="2.060301297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:00.057079645 +0000 UTC m=+2430.071641353" watchObservedRunningTime="2026-10-31 03:48:00.060301297 +0000 UTC m=+2430.074863005"
Oct 31 03:48:00.959270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:00.959234    5831 generic.go:356] "Generic (PLEG): container finished" podID="4c80aded-7843-40c3-b4cc-016fdb0b27fb" containerID="484d75138df3be136c7de7c859a3aca0b2f6f1ed9c0a7f70b38eb831b25d77de" exitCode=0
Oct 31 03:48:00.959714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:00.959331    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8" event={"ID":"4c80aded-7843-40c3-b4cc-016fdb0b27fb","Type":"ContainerDied","Data":"484d75138df3be136c7de7c859a3aca0b2f6f1ed9c0a7f70b38eb831b25d77de"}
Oct 31 03:48:02.228208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.228124    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:48:02.331472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.331438    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\") pod \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") "
Oct 31 03:48:02.331702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.331584    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\") pod \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") "
Oct 31 03:48:02.331787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.331761    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\" (UniqueName: \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\") pod \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\" (UID: \"4c80aded-7843-40c3-b4cc-016fdb0b27fb\") "
Oct 31 03:48:02.331850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.331811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca" pod "4c80aded-7843-40c3-b4cc-016fdb0b27fb" (UID: "4c80aded-7843-40c3-b4cc-016fdb0b27fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:02.332600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.332576    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:02.333993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.333969    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca" pod "4c80aded-7843-40c3-b4cc-016fdb0b27fb" (UID: "4c80aded-7843-40c3-b4cc-016fdb0b27fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:02.334107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.334094    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67" pod "4c80aded-7843-40c3-b4cc-016fdb0b27fb" (UID: "4c80aded-7843-40c3-b4cc-016fdb0b27fb"). InnerVolumeSpecName "kube-api-access-l9q67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:02.433298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.433259    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9q67\" (UniqueName: \"kubernetes.io/projected/4c80aded-7843-40c3-b4cc-016fdb0b27fb-kube-api-access-l9q67\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:02.433298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.433294    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c80aded-7843-40c3-b4cc-016fdb0b27fb-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:02.976702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.976669    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"
Oct 31 03:48:02.976702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.976683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8" event={"ID":"4c80aded-7843-40c3-b4cc-016fdb0b27fb","Type":"ContainerDied","Data":"f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b"}
Oct 31 03:48:02.976921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:02.976732    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9eb675dffe66f911b880644cafd4fadda2c3e78efb2d54716d6fb26cc96078b"
Oct 31 03:48:05.921975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:05.921943    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:48:05.922630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:05.922031    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:48:07.017235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.017201    5831 generic.go:356] "Generic (PLEG): container finished" podID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerID="efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781" exitCode=0
Oct 31 03:48:07.017718 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.017280    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerDied","Data":"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"}
Oct 31 03:48:07.017842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.017823    5831 scope.go:122] "RemoveContainer" containerID="efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"
Oct 31 03:48:07.019302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.019274    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:07.019432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.019358    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:07.019432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.019404    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:07.418058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:07.417973    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:48:08.027689 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.027647    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerStarted","Data":"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"}
Oct 31 03:48:08.534557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.512277    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"]
Oct 31 03:48:08.535821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.535786    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4c80aded-7843-40c3-b4cc-016fdb0b27fb" containerName="revoke-keys"
Oct 31 03:48:08.536019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.536004    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c80aded-7843-40c3-b4cc-016fdb0b27fb" containerName="revoke-keys"
Oct 31 03:48:08.536377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.536361    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c80aded-7843-40c3-b4cc-016fdb0b27fb" containerName="revoke-keys"
Oct 31 03:48:08.551555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.550811    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.599275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.599238    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"]
Oct 31 03:48:08.600769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.600481    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:08.600918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.600685    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:48:08.657928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.657882    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.658137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.657947    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.658137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.657999    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfsc\" (UniqueName: \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.759736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.759699    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.759736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.759742    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.759960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.759769    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfsc\" (UniqueName: \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.760470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.760440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.762567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.762521    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.812830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.812747    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfsc\" (UniqueName: \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\") pod \"maas-api-revoke-sub-824c927c-4fm6q\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:08.862073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.862037    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:48:08.862875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.862848    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 31 03:48:08.862940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.862909    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 31 03:48:08.904632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:08.900728    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:09.321646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:09.319843    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"]
Oct 31 03:48:09.322951 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:09.322903    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cf2aa4_497b_4e1b_a9e2_907574e9b2d8.slice/crio-d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08 WatchSource:0}: Error finding container d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08: Status 404 returned error can't find the container with id d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08
Oct 31 03:48:09.862921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:09.862885    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 31 03:48:09.863106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:09.862939    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 31 03:48:10.052350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:10.052244    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q" event={"ID":"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8","Type":"ContainerStarted","Data":"726241737eb81a1fe3dab5583816e32429ed8f9c0bc33f2104a90bdc0de7cf83"}
Oct 31 03:48:10.052350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:10.052314    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q" event={"ID":"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8","Type":"ContainerStarted","Data":"d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08"}
Oct 31 03:48:10.164232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:10.164181    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q" podStartSLOduration=2.164166749 podStartE2EDuration="2.164166749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:10.16271329 +0000 UTC m=+2440.177275008" watchObservedRunningTime="2026-10-31 03:48:10.164166749 +0000 UTC m=+2440.178728456"
Oct 31 03:48:10.861639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:10.861509    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 31 03:48:10.861639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:10.861627    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 31 03:48:11.064280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:11.064244    5831 generic.go:356] "Generic (PLEG): container finished" podID="f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" containerID="726241737eb81a1fe3dab5583816e32429ed8f9c0bc33f2104a90bdc0de7cf83" exitCode=0
Oct 31 03:48:11.064493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:11.064327    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q" event={"ID":"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8","Type":"ContainerDied","Data":"726241737eb81a1fe3dab5583816e32429ed8f9c0bc33f2104a90bdc0de7cf83"}
Oct 31 03:48:11.862173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:11.862137    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 31 03:48:11.862647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:11.862189    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 31 03:48:12.142327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.142238    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:48:12.337341 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.337017    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:12.431354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.430582    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\") pod \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") "
Oct 31 03:48:12.431354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.430651    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\" (UniqueName: \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\") pod \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") "
Oct 31 03:48:12.431354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.430684    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\") pod \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\" (UID: \"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8\") "
Oct 31 03:48:12.432233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.432191    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca" pod "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" (UID: "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:12.472555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.451276    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc" pod "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" (UID: "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8"). InnerVolumeSpecName "kube-api-access-jqfsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:12.525556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.474127    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca" pod "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" (UID: "f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:12.532336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.532297    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.532656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.532637    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqfsc\" (UniqueName: \"kubernetes.io/projected/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-kube-api-access-jqfsc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.532786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.532771    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.862369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.862291    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 31 03:48:12.862369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:12.862342    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 31 03:48:12.993866 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:12.993823    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:48:13.126365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:13.126276    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q" event={"ID":"f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8","Type":"ContainerDied","Data":"d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08"}
Oct 31 03:48:13.126574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:13.126378    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6fe39113ce3efd4d07b70f7873c756d227fa0eb4d79b45a446f91d4615dfd08"
Oct 31 03:48:13.126574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:13.126469    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"
Oct 31 03:48:13.668563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:13.666996    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:48:13.867065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:13.867027    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:48:14.144558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.142048    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:48:14.508897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.508860    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"]
Oct 31 03:48:14.509961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.509934    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" containerName="revoke-keys"
Oct 31 03:48:14.510146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.510116    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" containerName="revoke-keys"
Oct 31 03:48:14.510332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.510319    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" containerName="revoke-keys"
Oct 31 03:48:14.512717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.512698    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.561761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.561716    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:14.561761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.561733    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:48:14.566866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.566837    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"]
Oct 31 03:48:14.567548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.567511    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctqx\" (UniqueName: \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.567666 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.567652    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.567820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.567797    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.668420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.668379    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctqx\" (UniqueName: \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.668621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.668459    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.668621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.668571    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.669276 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.669248    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.671199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.671175    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.721655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.721620    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctqx\" (UniqueName: \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\") pod \"maas-api-revoke-sub-e3e47886-p5qnf\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:14.827399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:14.827313    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:15.203654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:15.203622    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"]
Oct 31 03:48:15.205358 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:15.205327    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546530ba_d570_4ff5_91b2_3c37a6c76067.slice/crio-0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21 WatchSource:0}: Error finding container 0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21: Status 404 returned error can't find the container with id 0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21
Oct 31 03:48:16.152340 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:16.152308    5831 generic.go:356] "Generic (PLEG): container finished" podID="546530ba-d570-4ff5-91b2-3c37a6c76067" containerID="305d065e6754f10681456bc2153b6da8a2f6c18da11ab81853b69017ddfa825c" exitCode=0
Oct 31 03:48:16.152555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:16.152392    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf" event={"ID":"546530ba-d570-4ff5-91b2-3c37a6c76067","Type":"ContainerDied","Data":"305d065e6754f10681456bc2153b6da8a2f6c18da11ab81853b69017ddfa825c"}
Oct 31 03:48:16.152555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:16.152445    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf" event={"ID":"546530ba-d570-4ff5-91b2-3c37a6c76067","Type":"ContainerStarted","Data":"0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21"}
Oct 31 03:48:17.143395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.143350    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" podUID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" containerName="payload-pre-processing" containerID="cri-o://291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4" gracePeriod=25
Oct 31 03:48:17.318313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.318285    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:17.400026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.399963    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\") pod \"546530ba-d570-4ff5-91b2-3c37a6c76067\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") "
Oct 31 03:48:17.400147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.400049    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\" (UniqueName: \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\") pod \"546530ba-d570-4ff5-91b2-3c37a6c76067\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") "
Oct 31 03:48:17.400147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.400072    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\") pod \"546530ba-d570-4ff5-91b2-3c37a6c76067\" (UID: \"546530ba-d570-4ff5-91b2-3c37a6c76067\") "
Oct 31 03:48:17.400480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.400453    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca" pod "546530ba-d570-4ff5-91b2-3c37a6c76067" (UID: "546530ba-d570-4ff5-91b2-3c37a6c76067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:17.402280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.402252    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca" pod "546530ba-d570-4ff5-91b2-3c37a6c76067" (UID: "546530ba-d570-4ff5-91b2-3c37a6c76067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:17.402371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.402356    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx" pod "546530ba-d570-4ff5-91b2-3c37a6c76067" (UID: "546530ba-d570-4ff5-91b2-3c37a6c76067"). InnerVolumeSpecName "kube-api-access-bctqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:17.423995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.423970    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:48:17.501608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.501575    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\") pod \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") "
Oct 31 03:48:17.501879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.501620    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\" (UniqueName: \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\") pod \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\" (UID: \"19f67d38-d3f3-4c15-8c6f-3ff689fe9625\") "
Oct 31 03:48:17.501972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.501892    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctqx\" (UniqueName: \"kubernetes.io/projected/546530ba-d570-4ff5-91b2-3c37a6c76067-kube-api-access-bctqx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:17.501972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.501908    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:17.501972 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.501920    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546530ba-d570-4ff5-91b2-3c37a6c76067-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:17.502108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.502017    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume" pod "19f67d38-d3f3-4c15-8c6f-3ff689fe9625" (UID: "19f67d38-d3f3-4c15-8c6f-3ff689fe9625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:17.504002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.503969    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm" pod "19f67d38-d3f3-4c15-8c6f-3ff689fe9625" (UID: "19f67d38-d3f3-4c15-8c6f-3ff689fe9625"). InnerVolumeSpecName "kube-api-access-jn4nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:17.603172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.603127    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:17.603172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:17.603166    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4nm\" (UniqueName: \"kubernetes.io/projected/19f67d38-d3f3-4c15-8c6f-3ff689fe9625-kube-api-access-jn4nm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:18.168958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.168928    5831 generic.go:356] "Generic (PLEG): container finished" podID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" containerID="291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4" exitCode=0
Oct 31 03:48:18.169331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.168980    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"
Oct 31 03:48:18.169331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.169003    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" event={"ID":"19f67d38-d3f3-4c15-8c6f-3ff689fe9625","Type":"ContainerDied","Data":"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"}
Oct 31 03:48:18.169331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.169051    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz" event={"ID":"19f67d38-d3f3-4c15-8c6f-3ff689fe9625","Type":"ContainerDied","Data":"9a9737ebfadaaccf3f7f6553c20287ef8590fb938780791aae3a9b3eb730c143"}
Oct 31 03:48:18.169331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.169076    5831 scope.go:122] "RemoveContainer" containerID="291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"
Oct 31 03:48:18.170739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.170721    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"
Oct 31 03:48:18.170870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.170759    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf" event={"ID":"546530ba-d570-4ff5-91b2-3c37a6c76067","Type":"ContainerDied","Data":"0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21"}
Oct 31 03:48:18.170870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.170806    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0564f89eae11bfe71f35238aac5307aac63544791fd3d6e60c5899f026328a21"
Oct 31 03:48:18.181178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.181160    5831 scope.go:122] "RemoveContainer" containerID="291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"
Oct 31 03:48:18.181471 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:18.181451    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4\": container with ID starting with 291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4 not found: ID does not exist" containerID="291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"
Oct 31 03:48:18.181522 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.181482    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4"} err="failed to get container status \"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4\": rpc error: code = NotFound desc = could not find container \"291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4\": container with ID starting with 291fff84a11844845cc6e22a9edaf2fc659c96dc557001d4c902f963e04114b4 not found: ID does not exist"
Oct 31 03:48:18.322002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.321967    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:48:18.368284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:18.368252    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dab9ee-697b86b4cf-rphxz"]
Oct 31 03:48:19.321252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.321218    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"]
Oct 31 03:48:19.321994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.321970    5831 cpu_manager.go:414] "containerMap: removing container" podUID="546530ba-d570-4ff5-91b2-3c37a6c76067" containerName="revoke-keys"
Oct 31 03:48:19.321994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.321994    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546530ba-d570-4ff5-91b2-3c37a6c76067" containerName="revoke-keys"
Oct 31 03:48:19.322175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.322033    5831 cpu_manager.go:414] "containerMap: removing container" podUID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" containerName="payload-pre-processing"
Oct 31 03:48:19.322175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.322039    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" containerName="payload-pre-processing"
Oct 31 03:48:19.322175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.322163    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" containerName="payload-pre-processing"
Oct 31 03:48:19.322175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.322176    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="546530ba-d570-4ff5-91b2-3c37a6c76067" containerName="revoke-keys"
Oct 31 03:48:19.326806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.326789    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.377877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.377847    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"]
Oct 31 03:48:19.418800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.418765    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:19.418998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.418915    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:48:19.423484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.423458    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8268\" (UniqueName: \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.423651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.423514    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.423696 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.423657    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.524715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.524673    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8268\" (UniqueName: \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.524980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.524753    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.524980 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.524822    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.525442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.525413    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.527630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.527595    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.577793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.577711    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8268\" (UniqueName: \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\") pod \"maas-api-revoke-sub-073e8d96-s4cnl\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:19.641508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:19.641474    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:20.015016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:20.014984    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"]
Oct 31 03:48:20.016634 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:20.016608    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ce95fe_a438_492b_b8b7_be170966f13b.slice/crio-0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e WatchSource:0}: Error finding container 0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e: Status 404 returned error can't find the container with id 0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e
Oct 31 03:48:20.136541 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:20.136481    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f67d38-d3f3-4c15-8c6f-3ff689fe9625" path="/var/lib/kubelet/pods/19f67d38-d3f3-4c15-8c6f-3ff689fe9625/volumes"
Oct 31 03:48:20.189681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:20.189646    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl" event={"ID":"d8ce95fe-a438-492b-b8b7-be170966f13b","Type":"ContainerStarted","Data":"0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e"}
Oct 31 03:48:21.200728 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:21.200642    5831 generic.go:356] "Generic (PLEG): container finished" podID="d8ce95fe-a438-492b-b8b7-be170966f13b" containerID="15d35375f26f7dc4b0fdf7e9daf541abe2852298208f02ad8ff04372a7e83392" exitCode=0
Oct 31 03:48:21.201159 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:21.200732    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl" event={"ID":"d8ce95fe-a438-492b-b8b7-be170966f13b","Type":"ContainerDied","Data":"15d35375f26f7dc4b0fdf7e9daf541abe2852298208f02ad8ff04372a7e83392"}
Oct 31 03:48:22.367062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.367030    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:22.457680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.457651    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\" (UniqueName: \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\") pod \"d8ce95fe-a438-492b-b8b7-be170966f13b\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") "
Oct 31 03:48:22.457920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.457755    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\") pod \"d8ce95fe-a438-492b-b8b7-be170966f13b\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") "
Oct 31 03:48:22.457920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.457910    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\") pod \"d8ce95fe-a438-492b-b8b7-be170966f13b\" (UID: \"d8ce95fe-a438-492b-b8b7-be170966f13b\") "
Oct 31 03:48:22.458183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.458159    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca" pod "d8ce95fe-a438-492b-b8b7-be170966f13b" (UID: "d8ce95fe-a438-492b-b8b7-be170966f13b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:22.458513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.458451    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:22.460165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.460138    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca" pod "d8ce95fe-a438-492b-b8b7-be170966f13b" (UID: "d8ce95fe-a438-492b-b8b7-be170966f13b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:22.460280 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.460244    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268" pod "d8ce95fe-a438-492b-b8b7-be170966f13b" (UID: "d8ce95fe-a438-492b-b8b7-be170966f13b"). InnerVolumeSpecName "kube-api-access-s8268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:22.560337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.560293    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8268\" (UniqueName: \"kubernetes.io/projected/d8ce95fe-a438-492b-b8b7-be170966f13b-kube-api-access-s8268\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:22.560337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:22.560326    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ce95fe-a438-492b-b8b7-be170966f13b-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:23.216802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:23.216767    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"
Oct 31 03:48:23.216802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:23.216799    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl" event={"ID":"d8ce95fe-a438-492b-b8b7-be170966f13b","Type":"ContainerDied","Data":"0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e"}
Oct 31 03:48:23.217066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:23.216899    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0e430e4303cfea6b8465a80847422a2833bbe91dff11531baa4a7e480d305e"
Oct 31 03:48:34.759787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:34.759752    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:48:34.760263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:34.759951    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy" containerID="cri-o://bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f" gracePeriod=30
Oct 31 03:48:35.977473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:35.977434    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"]
Oct 31 03:48:35.978315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:35.978295    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d8ce95fe-a438-492b-b8b7-be170966f13b" containerName="revoke-keys"
Oct 31 03:48:35.978374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:35.978318    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ce95fe-a438-492b-b8b7-be170966f13b" containerName="revoke-keys"
Oct 31 03:48:35.978432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:35.978423    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ce95fe-a438-492b-b8b7-be170966f13b" containerName="revoke-keys"
Oct 31 03:48:35.998849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:35.998815    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.036568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.036516    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"]
Oct 31 03:48:36.077346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.077309    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:36.077346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.077338    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:48:36.154349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.154314    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9685\" (UniqueName: \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.154349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.154348    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.154646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.154376    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.214668 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.214624    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"]
Oct 31 03:48:36.218992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.218967    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.256611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.256514    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9685\" (UniqueName: \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.256848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.256819    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.256967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.256865    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.257788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.257700    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.258041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.258019    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"]
Oct 31 03:48:36.259701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.259675    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.315856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.315820    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9685\" (UniqueName: \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\") pod \"maas-api-revoke-sub-0509fc0b-q62x6\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.317519 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.317496    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:36.358707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.358675    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76ns\" (UniqueName: \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.358858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.358798    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.358962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.358871    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.460240 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.460201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76ns\" (UniqueName: \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.460421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.460289    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.460421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.460376    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.461161 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.461133    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.462995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.462969    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.513682 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.513358    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76ns\" (UniqueName: \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\") pod \"maas-api-revoke-sub-a3fd9b5b-b56mn\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.534546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.534506    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:36.722118 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.722083    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"]
Oct 31 03:48:36.722389 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:36.722369    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd80bf2_1ae0_4b14_9873_c9cc15d1255e.slice/crio-31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563 WatchSource:0}: Error finding container 31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563: Status 404 returned error can't find the container with id 31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563
Oct 31 03:48:36.935079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:36.935051    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"]
Oct 31 03:48:36.936189 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:36.936166    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7ac748_67f4_49fd_95b6_3046ba44cd1f.slice/crio-76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad WatchSource:0}: Error finding container 76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad: Status 404 returned error can't find the container with id 76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad
Oct 31 03:48:37.348309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:37.348268    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn" event={"ID":"6d7ac748-67f4-49fd-95b6-3046ba44cd1f","Type":"ContainerStarted","Data":"76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad"}
Oct 31 03:48:37.349887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:37.349861    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6" event={"ID":"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e","Type":"ContainerStarted","Data":"e1a40132a55502b8cc7d82141958b48443c1b9ae3c53a8388fbfbf85fd7946ab"}
Oct 31 03:48:37.349964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:37.349916    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6" event={"ID":"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e","Type":"ContainerStarted","Data":"31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563"}
Oct 31 03:48:37.460117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:37.460058    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6" podStartSLOduration=2.460041579 podStartE2EDuration="2.460041579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:37.459623613 +0000 UTC m=+2467.474185326" watchObservedRunningTime="2026-10-31 03:48:37.460041579 +0000 UTC m=+2467.474603289"
Oct 31 03:48:38.359248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:38.359209    5831 generic.go:356] "Generic (PLEG): container finished" podID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" containerID="e1a40132a55502b8cc7d82141958b48443c1b9ae3c53a8388fbfbf85fd7946ab" exitCode=0
Oct 31 03:48:38.359736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:38.359301    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6" event={"ID":"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e","Type":"ContainerDied","Data":"e1a40132a55502b8cc7d82141958b48443c1b9ae3c53a8388fbfbf85fd7946ab"}
Oct 31 03:48:38.361082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:38.361056    5831 generic.go:356] "Generic (PLEG): container finished" podID="6d7ac748-67f4-49fd-95b6-3046ba44cd1f" containerID="171eaecc3e76efbab95b01c34257d59781f104931e334bbd1759f11b3470da08" exitCode=0
Oct 31 03:48:38.361216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:38.361135    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn" event={"ID":"6d7ac748-67f4-49fd-95b6-3046ba44cd1f","Type":"ContainerDied","Data":"171eaecc3e76efbab95b01c34257d59781f104931e334bbd1759f11b3470da08"}
Oct 31 03:48:39.630853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.630826    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:39.658905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.658878    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:39.726590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.726558    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\" (UniqueName: \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\") pod \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") "
Oct 31 03:48:39.726780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.726737    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\") pod \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") "
Oct 31 03:48:39.726780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.726769    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\") pod \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\" (UID: \"6d7ac748-67f4-49fd-95b6-3046ba44cd1f\") "
Oct 31 03:48:39.727171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.727140    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca" pod "6d7ac748-67f4-49fd-95b6-3046ba44cd1f" (UID: "6d7ac748-67f4-49fd-95b6-3046ba44cd1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:39.727459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.727233    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:39.728960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.728926    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns" pod "6d7ac748-67f4-49fd-95b6-3046ba44cd1f" (UID: "6d7ac748-67f4-49fd-95b6-3046ba44cd1f"). InnerVolumeSpecName "kube-api-access-k76ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:39.729064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.729016    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca" pod "6d7ac748-67f4-49fd-95b6-3046ba44cd1f" (UID: "6d7ac748-67f4-49fd-95b6-3046ba44cd1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:39.832042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.830938    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\") pod \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") "
Oct 31 03:48:39.832042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.830996    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\") pod \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") "
Oct 31 03:48:39.832042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.831184    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\" (UniqueName: \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\") pod \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\" (UID: \"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e\") "
Oct 31 03:48:39.832042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.831622    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:39.832042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.831642    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76ns\" (UniqueName: \"kubernetes.io/projected/6d7ac748-67f4-49fd-95b6-3046ba44cd1f-kube-api-access-k76ns\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:39.832927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.832883    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca" pod "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" (UID: "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:39.835367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.835320    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685" pod "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" (UID: "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e"). InnerVolumeSpecName "kube-api-access-k9685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:39.836893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.836857    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca" pod "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" (UID: "dfd80bf2-1ae0-4b14-9873-c9cc15d1255e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:39.933268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.933240    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:39.933268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.933270    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9685\" (UniqueName: \"kubernetes.io/projected/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-kube-api-access-k9685\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:39.933454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:39.933285    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.122470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.122444    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:48:40.222796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.222745    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:48:40.223477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223458    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" containerName="revoke-keys"
Oct 31 03:48:40.223685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223478    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" containerName="revoke-keys"
Oct 31 03:48:40.223685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223563    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6d7ac748-67f4-49fd-95b6-3046ba44cd1f" containerName="revoke-keys"
Oct 31 03:48:40.223685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223572    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7ac748-67f4-49fd-95b6-3046ba44cd1f" containerName="revoke-keys"
Oct 31 03:48:40.223685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223583    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy"
Oct 31 03:48:40.223685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223590    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy"
Oct 31 03:48:40.223989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223798    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" containerName="revoke-keys"
Oct 31 03:48:40.223989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223817    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7ac748-67f4-49fd-95b6-3046ba44cd1f" containerName="revoke-keys"
Oct 31 03:48:40.223989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.223827    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerName="istio-proxy"
Oct 31 03:48:40.227214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.227192    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.239651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239625    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.239801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239662    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.239801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239729    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.239801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239772    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.239801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239796    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.240041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239813    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.240041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239836    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.240041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.239929    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.240041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.240006    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\") pod \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\" (UID: \"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d\") "
Oct 31 03:48:40.240821 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.240655    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:40.240940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.240846    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:40.241184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.241009    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:40.241184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.241075    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:40.242901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.242877    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:40.243976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.243937    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:40.244583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.244557    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:40.244763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.244735    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "kube-api-access-dv44h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:40.246010 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.245988    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo" pod "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" (UID: "a74d7f77-bbea-4ca9-b6e4-c60078c63e7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:40.275801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.275755    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-de2877-serving-cert\""
Oct 31 03:48:40.276776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.276745    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:48:40.341631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341514    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.341841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341639    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgst\" (UniqueName: \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.341841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341781    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.341841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341836    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.342028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341974    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.341991    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342006    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342020    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv44h\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-kube-api-access-dv44h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342037    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342051    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342065    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342079    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.342231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.342094    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:40.376467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.376435    5831 generic.go:356] "Generic (PLEG): container finished" podID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" containerID="bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f" exitCode=0
Oct 31 03:48:40.376703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.376483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"
Oct 31 03:48:40.376703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.376515    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" event={"ID":"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d","Type":"ContainerDied","Data":"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"}
Oct 31 03:48:40.376703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.376608    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6" event={"ID":"a74d7f77-bbea-4ca9-b6e4-c60078c63e7d","Type":"ContainerDied","Data":"2fd8bed6f1ab9e61293221cbc016d3da6a748fe844e4a2e6df8cc15edb5c2051"}
Oct 31 03:48:40.376703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.376640    5831 scope.go:122] "RemoveContainer" containerID="bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"
Oct 31 03:48:40.378447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.378430    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"
Oct 31 03:48:40.378683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.378482    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn" event={"ID":"6d7ac748-67f4-49fd-95b6-3046ba44cd1f","Type":"ContainerDied","Data":"76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad"}
Oct 31 03:48:40.378683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.378521    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76031ef27c2b3768a2057016536b6c0c203d2495c959902011950417016de8ad"
Oct 31 03:48:40.414109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.414075    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:48:40.414109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.414095    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6" event={"ID":"dfd80bf2-1ae0-4b14-9873-c9cc15d1255e","Type":"ContainerDied","Data":"31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563"}
Oct 31 03:48:40.414333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.414144    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31668da21c261fa1c2d1e17fee60c5d13893c9fcf426fb11ff3070b4cbcd7563"
Oct 31 03:48:40.425823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.425803    5831 scope.go:122] "RemoveContainer" containerID="bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"
Oct 31 03:48:40.426172 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:40.426149    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f\": container with ID starting with bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f not found: ID does not exist" containerID="bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"
Oct 31 03:48:40.426258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.426184    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f"} err="failed to get container status \"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f\": rpc error: code = NotFound desc = could not find container \"bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f\": container with ID starting with bc606c06b888e129d381800310b596d2697075e1c6bdf9e60358de21aec4d16f not found: ID does not exist"
Oct 31 03:48:40.448555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.446425    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.448555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.446506    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.469555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.462097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.469555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.451191    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.469555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.462239    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgst\" (UniqueName: \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.469555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.462501    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.506613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.506504    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.516731 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.516695    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgst\" (UniqueName: \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\") pod \"maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.518031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.518004    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:48:40.523813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.523794    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.545952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.545926    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570705    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570820    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570867    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570898    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570936    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.570960    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rql\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.571000    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.571087    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.571142    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.575003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.573631    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.671999    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672044    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672094    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672139    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672165    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672183    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672205    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672220    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rql\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.672245    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.674992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.674177    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:48:40.677745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.676712    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.677745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.677391    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.677745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.677492    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.677990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.677787    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.678283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.678258    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zp2v6"]
Oct 31 03:48:40.678364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.678321    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.681078 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.681054    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.697352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.697325    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.730354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.730297    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.730765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.730740    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rql\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.837943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.837857    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:40.981945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:40.981909    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:48:40.991955 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:40.991914    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8c4660_090a_4eaf_b8a2_91de9e9032ec.slice/crio-bd966abbc80c8078864b0ec9d62fed17d7097746fe5844afb4cc5d03ab75f44a WatchSource:0}: Error finding container bd966abbc80c8078864b0ec9d62fed17d7097746fe5844afb4cc5d03ab75f44a: Status 404 returned error can't find the container with id bd966abbc80c8078864b0ec9d62fed17d7097746fe5844afb4cc5d03ab75f44a
Oct 31 03:48:41.224788 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:41.224737    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84b7ce9_cced_420b_89f6_b897f79430a0.slice/crio-8a3b18abec10dcb845b95379bda4a9983575b2a0fc98cf3f78bebe9cb86a70b0 WatchSource:0}: Error finding container 8a3b18abec10dcb845b95379bda4a9983575b2a0fc98cf3f78bebe9cb86a70b0: Status 404 returned error can't find the container with id 8a3b18abec10dcb845b95379bda4a9983575b2a0fc98cf3f78bebe9cb86a70b0
Oct 31 03:48:41.226521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.226482    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:48:41.228352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.228322    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:41.228461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.228384    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:41.228461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.228412    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:41.439605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.439564    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" event={"ID":"e84b7ce9-cced-420b-89f6-b897f79430a0","Type":"ContainerStarted","Data":"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"}
Oct 31 03:48:41.439890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.439871    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" event={"ID":"e84b7ce9-cced-420b-89f6-b897f79430a0","Type":"ContainerStarted","Data":"8a3b18abec10dcb845b95379bda4a9983575b2a0fc98cf3f78bebe9cb86a70b0"}
Oct 31 03:48:41.446579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.441478    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" event={"ID":"dc8c4660-090a-4eaf-b8a2-91de9e9032ec","Type":"ContainerStarted","Data":"bd966abbc80c8078864b0ec9d62fed17d7097746fe5844afb4cc5d03ab75f44a"}
Oct 31 03:48:41.549984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.549860    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podStartSLOduration=1.549841384 podStartE2EDuration="1.549841384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:41.548111326 +0000 UTC m=+2471.562673047" watchObservedRunningTime="2026-10-31 03:48:41.549841384 +0000 UTC m=+2471.564403094"
Oct 31 03:48:41.838429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:41.838354    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:42.137251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.137163    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74d7f77-bbea-4ca9-b6e4-c60078c63e7d" path="/var/lib/kubelet/pods/a74d7f77-bbea-4ca9-b6e4-c60078c63e7d/volumes"
Oct 31 03:48:42.451768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.451732    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" event={"ID":"dc8c4660-090a-4eaf-b8a2-91de9e9032ec","Type":"ContainerStarted","Data":"6f2eb0ba3c105b4e8b344878a0a1b1accdab294a3f3752a976dcba4394711bc3"}
Oct 31 03:48:42.451988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.451845    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:42.574595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.574520    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" podStartSLOduration=1.70687832 podStartE2EDuration="2.574506062s" totalImagesPullingTime="867.627742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:42.571504199 +0000 UTC m=+2472.586065912" watchObservedRunningTime="2026-10-31 03:48:42.574506062 +0000 UTC m=+2472.589067769"
Oct 31 03:48:42.840221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.840143    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:42.840221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:42.840204    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:42.986126 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:42.986023    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:48:43.838573 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:43.838510    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:43.838765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:43.838599    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:44.692553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.688763    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:48:44.692553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.688927    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:48:44.745016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.744978    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:48:44.745272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.745208    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" podUID="8cf79f28-966b-4b87-8261-dc71bd838c47" containerName="maas-api" containerID="cri-o://4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a" gracePeriod=30
Oct 31 03:48:44.844320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.844170    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:44.844320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:44.844248    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:45.338825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.338718    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:48:45.363050 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.363009    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") pod \"8cf79f28-966b-4b87-8261-dc71bd838c47\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") "
Oct 31 03:48:45.363183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.363058    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\") pod \"8cf79f28-966b-4b87-8261-dc71bd838c47\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") "
Oct 31 03:48:45.363183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.363132    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\") pod \"8cf79f28-966b-4b87-8261-dc71bd838c47\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") "
Oct 31 03:48:45.363183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.363167    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\" (UniqueName: \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\") pod \"8cf79f28-966b-4b87-8261-dc71bd838c47\" (UID: \"8cf79f28-966b-4b87-8261-dc71bd838c47\") "
Oct 31 03:48:45.363968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.363711    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca" pod "8cf79f28-966b-4b87-8261-dc71bd838c47" (UID: "8cf79f28-966b-4b87-8261-dc71bd838c47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:45.365873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.365844    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls" pod "8cf79f28-966b-4b87-8261-dc71bd838c47" (UID: "8cf79f28-966b-4b87-8261-dc71bd838c47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:45.366274 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.366243    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs" pod "8cf79f28-966b-4b87-8261-dc71bd838c47" (UID: "8cf79f28-966b-4b87-8261-dc71bd838c47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:45.366802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.366779    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6" pod "8cf79f28-966b-4b87-8261-dc71bd838c47" (UID: "8cf79f28-966b-4b87-8261-dc71bd838c47"). InnerVolumeSpecName "kube-api-access-4kdb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:45.467362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.465299    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:45.467362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.465336    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf79f28-966b-4b87-8261-dc71bd838c47-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:45.467362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.465350    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf79f28-966b-4b87-8261-dc71bd838c47-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:45.467362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.465365    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdb6\" (UniqueName: \"kubernetes.io/projected/8cf79f28-966b-4b87-8261-dc71bd838c47-kube-api-access-4kdb6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:45.492561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.489988    5831 generic.go:356] "Generic (PLEG): container finished" podID="8cf79f28-966b-4b87-8261-dc71bd838c47" containerID="4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a" exitCode=0
Oct 31 03:48:45.492561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.490160    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" event={"ID":"8cf79f28-966b-4b87-8261-dc71bd838c47","Type":"ContainerDied","Data":"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"}
Oct 31 03:48:45.492561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.490216    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl" event={"ID":"8cf79f28-966b-4b87-8261-dc71bd838c47","Type":"ContainerDied","Data":"b06eb6a74309e7bbda1e34a0c64f21ca55592dd41763a7b2957cb1fd042f2f90"}
Oct 31 03:48:45.492561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.490250    5831 scope.go:122] "RemoveContainer" containerID="4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"
Oct 31 03:48:45.492561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.490418    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"
Oct 31 03:48:45.617223 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.617196    5831 scope.go:122] "RemoveContainer" containerID="4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"
Oct 31 03:48:45.617671 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:45.617639    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a\": container with ID starting with 4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a not found: ID does not exist" containerID="4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"
Oct 31 03:48:45.617868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.617678    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a"} err="failed to get container status \"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a\": rpc error: code = NotFound desc = could not find container \"4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a\": container with ID starting with 4e4ddc2d82870e8377323aa083292e8da2d9c58d8310d871040272571470fa2a not found: ID does not exist"
Oct 31 03:48:45.708508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.708470    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:48:45.712177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.712149    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dab9ee-795778666d-2dzcl"]
Oct 31 03:48:45.839175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.839145    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:45.839362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:45.839206    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:46.135983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.135903    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf79f28-966b-4b87-8261-dc71bd838c47" path="/var/lib/kubelet/pods/8cf79f28-966b-4b87-8261-dc71bd838c47/volumes"
Oct 31 03:48:46.536912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.536349    5831 generic.go:356] "Generic (PLEG): container finished" podID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerID="7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0" exitCode=0
Oct 31 03:48:46.536912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.536436    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerDied","Data":"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"}
Oct 31 03:48:46.537191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.536966    5831 scope.go:122] "RemoveContainer" containerID="7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"
Oct 31 03:48:46.602385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.600566    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:46.602385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.600798    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:46.602385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.600906    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:48:46.839337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.839260    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:46.839786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:46.839625    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:47.546496 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:47.546449    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerStarted","Data":"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"}
Oct 31 03:48:47.838975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:47.838883    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 31 03:48:47.838975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:47.838963    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 31 03:48:48.409362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:48.408943    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:48:48.409362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:48.409069    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 31 03:48:48.409362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:48.409109    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 31 03:48:48.466790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:48.466761    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:48:49.413845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:49.412441    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 31 03:48:49.413845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:49.412542    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 31 03:48:49.448792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:49.448755    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:49.449237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:49.449218    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:49.450081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:49.450063    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:48:50.409017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:50.408979    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 31 03:48:50.409308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:50.409280    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 31 03:48:51.405994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:51.405954    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 31 03:48:51.406710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:51.406676    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 31 03:48:52.406557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:52.406109    5831 patch_prober.go:29] interesting pod/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 31 03:48:52.406557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:52.406189    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 31 03:48:52.552129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:52.552091    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:48:52.552369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:52.552344    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy" containerID="cri-o://5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48" gracePeriod=30
Oct 31 03:48:53.410503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.410466    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:48:53.411057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.410999    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:48:53.411894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.411869    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:48:53.893255 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.893218    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"]
Oct 31 03:48:53.893970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.893947    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8cf79f28-966b-4b87-8261-dc71bd838c47" containerName="maas-api"
Oct 31 03:48:53.893970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.893969    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf79f28-966b-4b87-8261-dc71bd838c47" containerName="maas-api"
Oct 31 03:48:53.894175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.894131    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf79f28-966b-4b87-8261-dc71bd838c47" containerName="maas-api"
Oct 31 03:48:53.898066 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.898044    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:53.946468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.946439    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:48:53.946685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.946516    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:53.949637 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:53.949517    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"]
Oct 31 03:48:54.036757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.036723    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.036990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.036881    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56j8\" (UniqueName: \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.037102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.037082    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.138470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.138431    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.138697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.138569    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56j8\" (UniqueName: \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.138771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.138692    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.139279 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.139251    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.141581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.141513    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.192304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.192268    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56j8\" (UniqueName: \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\") pod \"maas-api-revoke-sub-4f222b97-hzm89\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.212186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.212158    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:54.598918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.598886    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"]
Oct 31 03:48:54.599892 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:48:54.599862    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f559b11_bbbb_4ef5_9947_114c9476384e.slice/crio-f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3 WatchSource:0}: Error finding container f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3: Status 404 returned error can't find the container with id f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3
Oct 31 03:48:54.611979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:54.611944    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89" event={"ID":"6f559b11-bbbb-4ef5-9947-114c9476384e","Type":"ContainerStarted","Data":"f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3"}
Oct 31 03:48:55.622856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:55.622813    5831 generic.go:356] "Generic (PLEG): container finished" podID="6f559b11-bbbb-4ef5-9947-114c9476384e" containerID="5e0bc80d7dc47ff753f24f89691ced7a9081e6a2231a66491dc58b42f1c7b4e4" exitCode=0
Oct 31 03:48:55.623381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:55.622861    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89" event={"ID":"6f559b11-bbbb-4ef5-9947-114c9476384e","Type":"ContainerDied","Data":"5e0bc80d7dc47ff753f24f89691ced7a9081e6a2231a66491dc58b42f1c7b4e4"}
Oct 31 03:48:56.803168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.803140    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:56.977847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.977811    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\") pod \"6f559b11-bbbb-4ef5-9947-114c9476384e\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") "
Oct 31 03:48:56.978058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.977876    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\" (UniqueName: \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\") pod \"6f559b11-bbbb-4ef5-9947-114c9476384e\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") "
Oct 31 03:48:56.978058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.977904    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\") pod \"6f559b11-bbbb-4ef5-9947-114c9476384e\" (UID: \"6f559b11-bbbb-4ef5-9947-114c9476384e\") "
Oct 31 03:48:56.978172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.978150    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca" pod "6f559b11-bbbb-4ef5-9947-114c9476384e" (UID: "6f559b11-bbbb-4ef5-9947-114c9476384e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:56.978458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.978438    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.980297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.980270    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca" pod "6f559b11-bbbb-4ef5-9947-114c9476384e" (UID: "6f559b11-bbbb-4ef5-9947-114c9476384e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:56.980421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:56.980367    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8" pod "6f559b11-bbbb-4ef5-9947-114c9476384e" (UID: "6f559b11-bbbb-4ef5-9947-114c9476384e"). InnerVolumeSpecName "kube-api-access-d56j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:57.079657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.079622    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56j8\" (UniqueName: \"kubernetes.io/projected/6f559b11-bbbb-4ef5-9947-114c9476384e-kube-api-access-d56j8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:57.079657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.079657    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f559b11-bbbb-4ef5-9947-114c9476384e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:57.641793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.641758    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89" event={"ID":"6f559b11-bbbb-4ef5-9947-114c9476384e","Type":"ContainerDied","Data":"f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3"}
Oct 31 03:48:57.641977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.641815    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"
Oct 31 03:48:57.641977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.641822    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9dd7915f6ca569da93b7a59581f660e23633d83b604fb3236a3e624e43984c3"
Oct 31 03:48:57.852919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.852892    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:48:57.996015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.995973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996071    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996173    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996234    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996282    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996355 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996329    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996365    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996365    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:57.996546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996440    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996470    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\") pod \"08bf809b-e77c-4145-8949-b4935befb5e7\" (UID: \"08bf809b-e77c-4145-8949-b4935befb5e7\") "
Oct 31 03:48:57.996994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.996967    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:57.997239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.997179    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:57.997672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.997595    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:57.998095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.998076    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:57.998439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.998418    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:57.999339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.999313    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:57.999638 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:57.999614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:58.000942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.000921    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.001068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.001043    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt" pod "08bf809b-e77c-4145-8949-b4935befb5e7" (UID: "08bf809b-e77c-4145-8949-b4935befb5e7"). InnerVolumeSpecName "kube-api-access-wjwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098150    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098185    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwxt\" (UniqueName: \"kubernetes.io/projected/08bf809b-e77c-4145-8949-b4935befb5e7-kube-api-access-wjwxt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098198    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098209    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf809b-e77c-4145-8949-b4935befb5e7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098221    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf809b-e77c-4145-8949-b4935befb5e7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098233    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098244    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.098589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.098255    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf809b-e77c-4145-8949-b4935befb5e7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.651888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.651848    5831 generic.go:356] "Generic (PLEG): container finished" podID="08bf809b-e77c-4145-8949-b4935befb5e7" containerID="5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48" exitCode=0
Oct 31 03:48:58.652101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.651903    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"
Oct 31 03:48:58.652101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.651917    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerDied","Data":"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"}
Oct 31 03:48:58.652101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.651992    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k" event={"ID":"08bf809b-e77c-4145-8949-b4935befb5e7","Type":"ContainerDied","Data":"2841113e917a2955dad87d8506c49958da5f7edc53caa6f26f8b5f0f2367f21f"}
Oct 31 03:48:58.652101 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.652018    5831 scope.go:122] "RemoveContainer" containerID="5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"
Oct 31 03:48:58.663828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.663801    5831 scope.go:122] "RemoveContainer" containerID="a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"
Oct 31 03:48:58.674149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.674131    5831 scope.go:122] "RemoveContainer" containerID="5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"
Oct 31 03:48:58.674447 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:58.674427    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48\": container with ID starting with 5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48 not found: ID does not exist" containerID="5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"
Oct 31 03:48:58.674492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.674468    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48"} err="failed to get container status \"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48\": rpc error: code = NotFound desc = could not find container \"5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48\": container with ID starting with 5842e1bfdeef45d3621fe76ed0f5f09861e840de9a276f6f686c7c0c6ecdec48 not found: ID does not exist"
Oct 31 03:48:58.674492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.674489    5831 scope.go:122] "RemoveContainer" containerID="a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"
Oct 31 03:48:58.674851 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:48:58.674818    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2\": container with ID starting with a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2 not found: ID does not exist" containerID="a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"
Oct 31 03:48:58.674851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.674847    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2"} err="failed to get container status \"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2\": rpc error: code = NotFound desc = could not find container \"a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2\": container with ID starting with a281beed690d04ea7c7b9c7feeb99900d1a622e03ebc82eea8c6cad31eaef1e2 not found: ID does not exist"
Oct 31 03:48:58.807348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.807311    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:48:58.853180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:48:58.853139    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-dab9ee-openshift-default-744bfd9ffc-vpf8k"]
Oct 31 03:49:00.136760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:00.136726    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" path="/var/lib/kubelet/pods/08bf809b-e77c-4145-8949-b4935befb5e7/volumes"
Oct 31 03:49:05.144778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.144743    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:49:05.423583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.423440    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"]
Oct 31 03:49:05.424269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424245    5831 cpu_manager.go:414] "containerMap: removing container" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.424269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424268    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.424469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424287    5831 cpu_manager.go:414] "containerMap: removing container" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.424469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424294    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.424469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424361    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6f559b11-bbbb-4ef5-9947-114c9476384e" containerName="revoke-keys"
Oct 31 03:49:05.424469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424370    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f559b11-bbbb-4ef5-9947-114c9476384e" containerName="revoke-keys"
Oct 31 03:49:05.424721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424516    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f559b11-bbbb-4ef5-9947-114c9476384e" containerName="revoke-keys"
Oct 31 03:49:05.424721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424550    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.424721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.424562    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bf809b-e77c-4145-8949-b4935befb5e7" containerName="istio-proxy"
Oct 31 03:49:05.429582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.429562    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.481640 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.481608    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:49:05.491985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.491944    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.491985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.491995    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.492208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.492121    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz8b\" (UniqueName: \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.526402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.526373    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:49:05.530934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.530910    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"]
Oct 31 03:49:05.593106 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.593067    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.593325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.593120    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.593325 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.593210    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz8b\" (UniqueName: \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.594029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.594002    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.596293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.596271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.659820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.659787    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz8b\" (UniqueName: \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\") pod \"maas-api-revoke-sub-6e5a5fd3-n54mn\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:05.743810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:05.743775    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:06.222636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:06.222583    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"]
Oct 31 03:49:06.749186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:06.749148    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn" event={"ID":"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4","Type":"ContainerStarted","Data":"c3dcae359e180ec8db3f2967b2dc4caee01a324005fa4a890091aca2198b138c"}
Oct 31 03:49:07.758300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:07.758259    5831 generic.go:356] "Generic (PLEG): container finished" podID="5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" containerID="2d4822cd1e905504fcf486f4bd92c4ecdc3e503e13a05006d48fb61023f64ddb" exitCode=0
Oct 31 03:49:07.758823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:07.758335    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn" event={"ID":"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4","Type":"ContainerDied","Data":"2d4822cd1e905504fcf486f4bd92c4ecdc3e503e13a05006d48fb61023f64ddb"}
Oct 31 03:49:08.965855 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:08.965828    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:09.017090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.017005    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\") pod \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") "
Oct 31 03:49:09.017090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.017073    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\") pod \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") "
Oct 31 03:49:09.017295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.017157    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\" (UniqueName: \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\") pod \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\" (UID: \"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4\") "
Oct 31 03:49:09.017471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.017430    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca" pod "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" (UID: "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:09.017655 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.017639    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:09.019493 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.019469    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca" pod "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" (UID: "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:09.019595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.019561    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b" pod "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" (UID: "5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4"). InnerVolumeSpecName "kube-api-access-4lz8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:09.121660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.121621    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:09.121919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.121901    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lz8b\" (UniqueName: \"kubernetes.io/projected/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4-kube-api-access-4lz8b\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:09.775003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.774969    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"
Oct 31 03:49:09.775003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.774998    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn" event={"ID":"5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4","Type":"ContainerDied","Data":"c3dcae359e180ec8db3f2967b2dc4caee01a324005fa4a890091aca2198b138c"}
Oct 31 03:49:09.775252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:09.775049    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3dcae359e180ec8db3f2967b2dc4caee01a324005fa4a890091aca2198b138c"
Oct 31 03:49:12.996234 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:12.995935    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:49:15.823031 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.822990    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:15.823490 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.823042    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:15.831856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.831827    5831 generic.go:356] "Generic (PLEG): container finished" podID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerID="9601d6dd5267e5ccf56596a978437563d1a4f3ad66cf5763d03560dcfadbc5a8" exitCode=0
Oct 31 03:49:15.831990 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.831877    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerDied","Data":"9601d6dd5267e5ccf56596a978437563d1a4f3ad66cf5763d03560dcfadbc5a8"}
Oct 31 03:49:15.832261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.832248    5831 scope.go:122] "RemoveContainer" containerID="9601d6dd5267e5ccf56596a978437563d1a4f3ad66cf5763d03560dcfadbc5a8"
Oct 31 03:49:15.833731 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.833696    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:15.833800 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.833774    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:15.833843 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:15.833807    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:16.841403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:16.841370    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerStarted","Data":"c563e455284ae145038b05f1e6c5c4009476e89564ef39a12c9f0abfc4bf9db7"}
Oct 31 03:49:16.932023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:16.931993    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:49:16.932241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:16.932222    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:16.932289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:16.932264    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:17.932369 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:17.932326    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:17.932833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:17.932397    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:18.932361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:18.932326    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:18.932630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:18.932391    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:19.932365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:19.932330    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:19.932593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:19.932401    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:20.932354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:20.932313    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:20.932788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:20.932368    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:21.062739 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.062705    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"]
Oct 31 03:49:21.063431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.063411    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" containerName="revoke-keys"
Oct 31 03:49:21.063431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.063432    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" containerName="revoke-keys"
Oct 31 03:49:21.063665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.063591    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" containerName="revoke-keys"
Oct 31 03:49:21.067475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.067457    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.116516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.116482    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:49:21.121511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.121477    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"]
Oct 31 03:49:21.160875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.160842    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:49:21.175582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.175554    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.176003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.175981    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ssb\" (UniqueName: \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.176305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.176267    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.277832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.277744    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.277832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.277827    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ssb\" (UniqueName: \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.278199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.278180    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.285550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.285069    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.310289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.310220    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.331104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.331074    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ssb\" (UniqueName: \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\") pod \"maas-api-revoke-sub-ee846898-wnjvn\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.392995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.392958    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:21.801602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.801567    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"]
Oct 31 03:49:21.801789 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:21.801643    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4227d42_b4a7_4e92_9100_dc339a18c62e.slice/crio-13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc WatchSource:0}: Error finding container 13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc: Status 404 returned error can't find the container with id 13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc
Oct 31 03:49:21.879974 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.879930    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn" event={"ID":"a4227d42-b4a7-4e92-9100-dc339a18c62e","Type":"ContainerStarted","Data":"13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc"}
Oct 31 03:49:21.932675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.932631    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:49:21.933079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:21.932706    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:49:22.890018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:22.889976    5831 generic.go:356] "Generic (PLEG): container finished" podID="a4227d42-b4a7-4e92-9100-dc339a18c62e" containerID="c0a8ef76988fcc659c176079bba81d862d535177b60b252973237454d1b98b68" exitCode=0
Oct 31 03:49:22.890201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:22.890070    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn" event={"ID":"a4227d42-b4a7-4e92-9100-dc339a18c62e","Type":"ContainerDied","Data":"c0a8ef76988fcc659c176079bba81d862d535177b60b252973237454d1b98b68"}
Oct 31 03:49:22.937160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:22.937129    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:49:22.937652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:22.937454    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:49:22.938394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:22.938369    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:49:23.387126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:23.387089    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:49:23.487079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:23.487046    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:49:24.117153 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.117121    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:24.256505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.256419    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\") pod \"a4227d42-b4a7-4e92-9100-dc339a18c62e\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") "
Oct 31 03:49:24.256505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.256468    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\") pod \"a4227d42-b4a7-4e92-9100-dc339a18c62e\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") "
Oct 31 03:49:24.256786 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.256641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\" (UniqueName: \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\") pod \"a4227d42-b4a7-4e92-9100-dc339a18c62e\" (UID: \"a4227d42-b4a7-4e92-9100-dc339a18c62e\") "
Oct 31 03:49:24.256881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.256856    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca" pod "a4227d42-b4a7-4e92-9100-dc339a18c62e" (UID: "a4227d42-b4a7-4e92-9100-dc339a18c62e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:24.257263 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.257243    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:24.259434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.259399    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca" pod "a4227d42-b4a7-4e92-9100-dc339a18c62e" (UID: "a4227d42-b4a7-4e92-9100-dc339a18c62e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:24.259604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.259471    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb" pod "a4227d42-b4a7-4e92-9100-dc339a18c62e" (UID: "a4227d42-b4a7-4e92-9100-dc339a18c62e"). InnerVolumeSpecName "kube-api-access-g8ssb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:24.359295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.359260    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4227d42-b4a7-4e92-9100-dc339a18c62e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:24.359295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.359291    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ssb\" (UniqueName: \"kubernetes.io/projected/a4227d42-b4a7-4e92-9100-dc339a18c62e-kube-api-access-g8ssb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:24.955236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.955190    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn" event={"ID":"a4227d42-b4a7-4e92-9100-dc339a18c62e","Type":"ContainerDied","Data":"13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc"}
Oct 31 03:49:24.955236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.955218    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"
Oct 31 03:49:24.955516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:24.955265    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13570b7a9271f3c83fdc3baccd0bb089116b2af535bf5e3f0135fc5e988898bc"
Oct 31 03:49:28.387789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.387698    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" podUID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" containerName="payload-processing" containerID="cri-o://3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0" gracePeriod=25
Oct 31 03:49:28.488408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.488358    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" podUID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" containerName="payload-pre-processing" containerID="cri-o://86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530" gracePeriod=25
Oct 31 03:49:28.672418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.672392    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:49:28.710763 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.710729    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\" (UniqueName: \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\") pod \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") "
Oct 31 03:49:28.710946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.710859    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\") pod \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\" (UID: \"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1\") "
Oct 31 03:49:28.711674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.711643    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume" pod "8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" (UID: "8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:28.714740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.714083    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v" pod "8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" (UID: "8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1"). InnerVolumeSpecName "kube-api-access-48x8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:28.768772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.768745    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:49:28.811853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.811813    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\" (UniqueName: \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\") pod \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") "
Oct 31 03:49:28.812056 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.811891    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\") pod \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\" (UID: \"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7\") "
Oct 31 03:49:28.812320 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.812293    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume" pod "52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" (UID: "52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:28.812446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.812374    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:28.812446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.812397    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48x8v\" (UniqueName: \"kubernetes.io/projected/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1-kube-api-access-48x8v\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:28.814811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.814788    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5" pod "52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" (UID: "52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7"). InnerVolumeSpecName "kube-api-access-wbkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:28.913796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.913701    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkj5\" (UniqueName: \"kubernetes.io/projected/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-kube-api-access-wbkj5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:28.913796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.913750    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:28.986321 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.986284    5831 generic.go:356] "Generic (PLEG): container finished" podID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" containerID="3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0" exitCode=0
Oct 31 03:49:28.986486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.986333    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"
Oct 31 03:49:28.986486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.986373    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" event={"ID":"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1","Type":"ContainerDied","Data":"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"}
Oct 31 03:49:28.986486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.986426    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm" event={"ID":"8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1","Type":"ContainerDied","Data":"3c3697e2b1926d1880b01a961891195a9d8d98a8a13501bf5122ee9419bd9747"}
Oct 31 03:49:28.986486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.986453    5831 scope.go:122] "RemoveContainer" containerID="3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"
Oct 31 03:49:28.988247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.988221    5831 generic.go:356] "Generic (PLEG): container finished" podID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" containerID="86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530" exitCode=0
Oct 31 03:49:28.988372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.988355    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" event={"ID":"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7","Type":"ContainerDied","Data":"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"}
Oct 31 03:49:28.988425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.988396    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4" event={"ID":"52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7","Type":"ContainerDied","Data":"232c6798e9277dee14da7319b9a190bb1034fc2d90c8507d82107e6d088eeae9"}
Oct 31 03:49:28.988471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.988462    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"
Oct 31 03:49:28.998932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.998910    5831 scope.go:122] "RemoveContainer" containerID="3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"
Oct 31 03:49:28.999330 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:28.999292    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0\": container with ID starting with 3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0 not found: ID does not exist" containerID="3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"
Oct 31 03:49:28.999416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.999334    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0"} err="failed to get container status \"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0\": rpc error: code = NotFound desc = could not find container \"3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0\": container with ID starting with 3d9951a074f2003742c3f2594207ec2ae79a4378a2ea1c27f36c71ee606761a0 not found: ID does not exist"
Oct 31 03:49:28.999416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:28.999360    5831 scope.go:122] "RemoveContainer" containerID="86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"
Oct 31 03:49:29.009565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.009523    5831 scope.go:122] "RemoveContainer" containerID="86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"
Oct 31 03:49:29.009866 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:29.009842    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530\": container with ID starting with 86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530 not found: ID does not exist" containerID="86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"
Oct 31 03:49:29.009932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.009874    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530"} err="failed to get container status \"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530\": rpc error: code = NotFound desc = could not find container \"86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530\": container with ID starting with 86e4b034839a2eb1e1c422b378d0d7ab1bae37d3c6fc9d15f8275726bc3cd530 not found: ID does not exist"
Oct 31 03:49:29.141902 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.141866    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:49:29.145498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.145468    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-54c955d64c-lh7nm"]
Oct 31 03:49:29.201632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.201602    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 03:49:29.202191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202177    5831 cpu_manager.go:414] "containerMap: removing container" podUID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" containerName="payload-pre-processing"
Oct 31 03:49:29.202243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202192    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" containerName="payload-pre-processing"
Oct 31 03:49:29.202243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202215    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a4227d42-b4a7-4e92-9100-dc339a18c62e" containerName="revoke-keys"
Oct 31 03:49:29.202243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202221    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4227d42-b4a7-4e92-9100-dc339a18c62e" containerName="revoke-keys"
Oct 31 03:49:29.202243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202232    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" containerName="payload-processing"
Oct 31 03:49:29.202243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202239    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" containerName="payload-processing"
Oct 31 03:49:29.202428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202335    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" containerName="payload-pre-processing"
Oct 31 03:49:29.202428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202349    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4227d42-b4a7-4e92-9100-dc339a18c62e" containerName="revoke-keys"
Oct 31 03:49:29.202428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.202360    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" containerName="payload-processing"
Oct 31 03:49:29.207226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.207207    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217468    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217609    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217659    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217885    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217917    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.217901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.217979    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.218316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.218283    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.218431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.218395    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmdh\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.218508 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.218434    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.247315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.247286    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:49:29.252038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.252010    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-6bb76ffd95-v8fq4"]
Oct 31 03:49:29.257324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.257302    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d39257-openshift-default-dockercfg-bm5z5\""
Oct 31 03:49:29.262134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.261991    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 03:49:29.319190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319148    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmdh\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319190 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319189    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319280    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319330    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319361    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319413    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319440    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.319810 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.319516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.320197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.320135    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.320197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.320155    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.320388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.320369    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.320486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.320466    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.320571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.320482    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.322735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.322713    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.322856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.322839    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.382155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.382116    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmdh\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.382315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.382177    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.522189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.522090    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:29.949239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.949206    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 03:49:29.950338 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:29.950312    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb36c39_f27d_4902_b7da_184bf3641495.slice/crio-4af15c18e77f70d7ba8d1f5f38c4f7c03eebac48a74f7f291f1c3e2f136c644a WatchSource:0}: Error finding container 4af15c18e77f70d7ba8d1f5f38c4f7c03eebac48a74f7f291f1c3e2f136c644a: Status 404 returned error can't find the container with id 4af15c18e77f70d7ba8d1f5f38c4f7c03eebac48a74f7f291f1c3e2f136c644a
Oct 31 03:49:29.953188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.953148    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:29.953285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.953238    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:29.953285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:29.953279    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:30.000216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:30.000177    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" event={"ID":"feb36c39-f27d-4902-b7da-184bf3641495","Type":"ContainerStarted","Data":"4af15c18e77f70d7ba8d1f5f38c4f7c03eebac48a74f7f291f1c3e2f136c644a"}
Oct 31 03:49:30.158003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:30.157962    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7" path="/var/lib/kubelet/pods/52c506e7-8b43-4bcf-8c2d-2fdc1079a6b7/volumes"
Oct 31 03:49:30.163171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:30.163129    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1" path="/var/lib/kubelet/pods/8a26d0ee-10bf-4dd1-9580-4bbce4ace5c1/volumes"
Oct 31 03:49:31.015714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:31.015673    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" event={"ID":"feb36c39-f27d-4902-b7da-184bf3641495","Type":"ContainerStarted","Data":"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"}
Oct 31 03:49:31.239040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:31.238978    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podStartSLOduration=2.238962605 podStartE2EDuration="2.238962605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:31.190139566 +0000 UTC m=+2521.204701275" watchObservedRunningTime="2026-10-31 03:49:31.238962605 +0000 UTC m=+2521.253524313"
Oct 31 03:49:31.522798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:31.522714    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:31.525253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:31.525222    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:31.525503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:31.525479    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:32.523556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:32.522962    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:32.523556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:32.523029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:33.522553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:33.522501    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:33.522742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:33.522588    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:34.342297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.342262    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"]
Oct 31 03:49:34.356098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.356062    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.413947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.413910    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"]
Oct 31 03:49:34.424250 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.424216    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lk2\" (UniqueName: \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.424451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.424416    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.424613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.424590    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.453746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.453694    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:49:34.453746 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.453704    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:49:34.522906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.522868    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:34.523088 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.522935    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:34.525438 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.525408    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.525590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.525500    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.525590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.525572    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lk2\" (UniqueName: \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.526121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.526098    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.528019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.527994    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.578459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.578426    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lk2\" (UniqueName: \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\") pod \"maas-api-revoke-sub-4bf83654-ws9g8\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:34.674809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:34.674725    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:35.108793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:35.107206    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"]
Oct 31 03:49:35.207557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:35.191742    5831 scope.go:122] "RemoveContainer" containerID="3b9efe4c8071ab35dcf8553281638ff9a48cb0fd329563a0b07f13a30811a049"
Oct 31 03:49:35.522642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:35.522608    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:35.523044 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:35.522668    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:36.082126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:36.082088    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8" event={"ID":"2426bd03-8970-451b-846d-5f5813e737e2","Type":"ContainerStarted","Data":"cee54058afdcdb54cdc87a41eb815258b583ea6fed4521b1fd7a2ab08a768a7b"}
Oct 31 03:49:36.082715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:36.082147    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8" event={"ID":"2426bd03-8970-451b-846d-5f5813e737e2","Type":"ContainerStarted","Data":"5ecfdf9bcc2e96033062ca26a5a6ed83e4477001f2ca2ad4f1a3fdcbd7978706"}
Oct 31 03:49:36.192669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:36.192615    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8" podStartSLOduration=2.192602255 podStartE2EDuration="2.192602255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:36.189559185 +0000 UTC m=+2526.204120891" watchObservedRunningTime="2026-10-31 03:49:36.192602255 +0000 UTC m=+2526.207163963"
Oct 31 03:49:36.523342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:36.523305    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 31 03:49:36.523760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:36.523358    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 31 03:49:37.093054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:37.093014    5831 generic.go:356] "Generic (PLEG): container finished" podID="2426bd03-8970-451b-846d-5f5813e737e2" containerID="cee54058afdcdb54cdc87a41eb815258b583ea6fed4521b1fd7a2ab08a768a7b" exitCode=0
Oct 31 03:49:37.093267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:37.093067    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8" event={"ID":"2426bd03-8970-451b-846d-5f5813e737e2","Type":"ContainerDied","Data":"cee54058afdcdb54cdc87a41eb815258b583ea6fed4521b1fd7a2ab08a768a7b"}
Oct 31 03:49:38.298254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.298217    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:38.446053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.446022    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\" (UniqueName: \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\") pod \"2426bd03-8970-451b-846d-5f5813e737e2\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") "
Oct 31 03:49:38.446416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.446384    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\") pod \"2426bd03-8970-451b-846d-5f5813e737e2\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") "
Oct 31 03:49:38.446603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.446563    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\") pod \"2426bd03-8970-451b-846d-5f5813e737e2\" (UID: \"2426bd03-8970-451b-846d-5f5813e737e2\") "
Oct 31 03:49:38.447024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.446945    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca" pod "2426bd03-8970-451b-846d-5f5813e737e2" (UID: "2426bd03-8970-451b-846d-5f5813e737e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:38.455704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.449322    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2" pod "2426bd03-8970-451b-846d-5f5813e737e2" (UID: "2426bd03-8970-451b-846d-5f5813e737e2"). InnerVolumeSpecName "kube-api-access-x2lk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:38.455704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.449329    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca" pod "2426bd03-8970-451b-846d-5f5813e737e2" (UID: "2426bd03-8970-451b-846d-5f5813e737e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:38.533093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.533050    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:49:38.533298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.533125    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:49:38.548324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.548284    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lk2\" (UniqueName: \"kubernetes.io/projected/2426bd03-8970-451b-846d-5f5813e737e2-kube-api-access-x2lk2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:38.548324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.548320    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:38.548324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.548331    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2426bd03-8970-451b-846d-5f5813e737e2-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:38.614224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.614193    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:38.614717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.614701    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:38.615817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:38.615793    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 03:49:39.115515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.115480    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"
Oct 31 03:49:39.115515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.115495    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8" event={"ID":"2426bd03-8970-451b-846d-5f5813e737e2","Type":"ContainerDied","Data":"5ecfdf9bcc2e96033062ca26a5a6ed83e4477001f2ca2ad4f1a3fdcbd7978706"}
Oct 31 03:49:39.115832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.115569    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ecfdf9bcc2e96033062ca26a5a6ed83e4477001f2ca2ad4f1a3fdcbd7978706"
Oct 31 03:49:39.285165 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.285127    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:49:39.285439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.285415    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" podUID="2c28d428-0962-4010-b640-3b601f53297e" containerName="maas-api" containerID="cri-o://c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa" gracePeriod=30
Oct 31 03:49:39.654553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.654460    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:49:39.694590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.683951    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\") pod \"2c28d428-0962-4010-b640-3b601f53297e\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") "
Oct 31 03:49:39.694590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.684041    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") pod \"2c28d428-0962-4010-b640-3b601f53297e\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") "
Oct 31 03:49:39.694590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.684081    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\") pod \"2c28d428-0962-4010-b640-3b601f53297e\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") "
Oct 31 03:49:39.694590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.684202    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\" (UniqueName: \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\") pod \"2c28d428-0962-4010-b640-3b601f53297e\" (UID: \"2c28d428-0962-4010-b640-3b601f53297e\") "
Oct 31 03:49:39.694590 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.688440    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca" pod "2c28d428-0962-4010-b640-3b601f53297e" (UID: "2c28d428-0962-4010-b640-3b601f53297e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:39.719069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.719008    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg" pod "2c28d428-0962-4010-b640-3b601f53297e" (UID: "2c28d428-0962-4010-b640-3b601f53297e"). InnerVolumeSpecName "kube-api-access-rk6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:39.739813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.739779    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls" pod "2c28d428-0962-4010-b640-3b601f53297e" (UID: "2c28d428-0962-4010-b640-3b601f53297e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:39.755674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.755616    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs" pod "2c28d428-0962-4010-b640-3b601f53297e" (UID: "2c28d428-0962-4010-b640-3b601f53297e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:39.786311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.786269    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:39.786311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.786314    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28d428-0962-4010-b640-3b601f53297e-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:39.786612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.786333    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6wg\" (UniqueName: \"kubernetes.io/projected/2c28d428-0962-4010-b640-3b601f53297e-kube-api-access-rk6wg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:39.786612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:39.786348    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28d428-0962-4010-b640-3b601f53297e-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.124950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.124908    5831 generic.go:356] "Generic (PLEG): container finished" podID="2c28d428-0962-4010-b640-3b601f53297e" containerID="c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa" exitCode=0
Oct 31 03:49:40.124950 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.124956    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"
Oct 31 03:49:40.125244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.124997    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" event={"ID":"2c28d428-0962-4010-b640-3b601f53297e","Type":"ContainerDied","Data":"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"}
Oct 31 03:49:40.125244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.125051    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8" event={"ID":"2c28d428-0962-4010-b640-3b601f53297e","Type":"ContainerDied","Data":"64bf435a828360343fe4b85bcfc257d21483bf9f596374d4af1ceec59ae34493"}
Oct 31 03:49:40.125244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.125072    5831 scope.go:122] "RemoveContainer" containerID="c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"
Oct 31 03:49:40.137039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.137017    5831 scope.go:122] "RemoveContainer" containerID="c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"
Oct 31 03:49:40.137314 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:40.137298    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa\": container with ID starting with c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa not found: ID does not exist" containerID="c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"
Oct 31 03:49:40.137399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.137321    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa"} err="failed to get container status \"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa\": rpc error: code = NotFound desc = could not find container \"c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa\": container with ID starting with c54121e5286b605bd5e9888b794de2f91b9bd301b96b246b30b854ae9ed6b6fa not found: ID does not exist"
Oct 31 03:49:40.297845 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.297806    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:49:40.372551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:40.372140    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc85c7-6cd94cdc85-26sn8"]
Oct 31 03:49:42.136378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:42.136337    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c28d428-0962-4010-b640-3b601f53297e" path="/var/lib/kubelet/pods/2c28d428-0962-4010-b640-3b601f53297e/volumes"
Oct 31 03:49:42.981638 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:42.981601    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:49:44.563629 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.563589    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"]
Oct 31 03:49:44.564383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564359    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2c28d428-0962-4010-b640-3b601f53297e" containerName="maas-api"
Oct 31 03:49:44.564383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564384    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c28d428-0962-4010-b640-3b601f53297e" containerName="maas-api"
Oct 31 03:49:44.564594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564422    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2426bd03-8970-451b-846d-5f5813e737e2" containerName="revoke-keys"
Oct 31 03:49:44.564594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564427    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2426bd03-8970-451b-846d-5f5813e737e2" containerName="revoke-keys"
Oct 31 03:49:44.564594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564542    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2426bd03-8970-451b-846d-5f5813e737e2" containerName="revoke-keys"
Oct 31 03:49:44.564594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.564562    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c28d428-0962-4010-b640-3b601f53297e" containerName="maas-api"
Oct 31 03:49:44.568034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.568019    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.622178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.622138    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"]
Oct 31 03:49:44.660217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.660174    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mmdf\" (UniqueName: \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.660430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.660289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.660430 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.660335    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.661750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.661723    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:49:44.661887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.661809    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:49:44.762017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.761983    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mmdf\" (UniqueName: \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.762180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.762074    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.762180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.762107    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.762796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.762772    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.764760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.764735    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.815481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.815409    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mmdf\" (UniqueName: \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\") pod \"maas-api-revoke-sub-425dc93c-v94r2\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:44.882121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:44.882085    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:45.257239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:45.257209    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"]
Oct 31 03:49:45.258433 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:45.258407    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cac73b5_4073_48b9_a4e8_6d12264ca322.slice/crio-f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb WatchSource:0}: Error finding container f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb: Status 404 returned error can't find the container with id f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb
Oct 31 03:49:46.176338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:46.176299    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2" event={"ID":"6cac73b5-4073-48b9-a4e8-6d12264ca322","Type":"ContainerStarted","Data":"9b818a297aab94651c61961b84b949276de579300c9356ecabde9ca9ca0bff46"}
Oct 31 03:49:46.176850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:46.176351    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2" event={"ID":"6cac73b5-4073-48b9-a4e8-6d12264ca322","Type":"ContainerStarted","Data":"f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb"}
Oct 31 03:49:46.337107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:46.334794    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2" podStartSLOduration=2.334774445 podStartE2EDuration="2.334774445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:46.332593206 +0000 UTC m=+2536.347154914" watchObservedRunningTime="2026-10-31 03:49:46.334774445 +0000 UTC m=+2536.349336154"
Oct 31 03:49:47.194043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:47.193996    5831 generic.go:356] "Generic (PLEG): container finished" podID="6cac73b5-4073-48b9-a4e8-6d12264ca322" containerID="9b818a297aab94651c61961b84b949276de579300c9356ecabde9ca9ca0bff46" exitCode=0
Oct 31 03:49:47.194575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:47.194132    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2" event={"ID":"6cac73b5-4073-48b9-a4e8-6d12264ca322","Type":"ContainerDied","Data":"9b818a297aab94651c61961b84b949276de579300c9356ecabde9ca9ca0bff46"}
Oct 31 03:49:47.695764 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:47.695731    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:49:47.696196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:47.696166    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy" containerID="cri-o://0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a" gracePeriod=30
Oct 31 03:49:48.362568 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.362513    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:48.460272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.460188    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\" (UniqueName: \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\") pod \"6cac73b5-4073-48b9-a4e8-6d12264ca322\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") "
Oct 31 03:49:48.460272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.460238    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\") pod \"6cac73b5-4073-48b9-a4e8-6d12264ca322\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") "
Oct 31 03:49:48.460474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.460287    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\") pod \"6cac73b5-4073-48b9-a4e8-6d12264ca322\" (UID: \"6cac73b5-4073-48b9-a4e8-6d12264ca322\") "
Oct 31 03:49:48.460730 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.460702    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca" pod "6cac73b5-4073-48b9-a4e8-6d12264ca322" (UID: "6cac73b5-4073-48b9-a4e8-6d12264ca322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:48.462601 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.462574    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca" pod "6cac73b5-4073-48b9-a4e8-6d12264ca322" (UID: "6cac73b5-4073-48b9-a4e8-6d12264ca322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:48.462820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.462796    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf" pod "6cac73b5-4073-48b9-a4e8-6d12264ca322" (UID: "6cac73b5-4073-48b9-a4e8-6d12264ca322"). InnerVolumeSpecName "kube-api-access-6mmdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:48.561213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.561175    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mmdf\" (UniqueName: \"kubernetes.io/projected/6cac73b5-4073-48b9-a4e8-6d12264ca322-kube-api-access-6mmdf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:48.561213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.561207    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:48.561213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.561219    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cac73b5-4073-48b9-a4e8-6d12264ca322-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:48.709202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:48.709169    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:49:49.210268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.210232    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"
Oct 31 03:49:49.210268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.210251    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2" event={"ID":"6cac73b5-4073-48b9-a4e8-6d12264ca322","Type":"ContainerDied","Data":"f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb"}
Oct 31 03:49:49.210495 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.210295    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3f4d07fbf15fa212707e0f0e4df03e1150430c80d75d89224aab72b5e40a2bb"
Oct 31 03:49:49.696074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.696040    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"]
Oct 31 03:49:49.696703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.696688    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6cac73b5-4073-48b9-a4e8-6d12264ca322" containerName="revoke-keys"
Oct 31 03:49:49.696750 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.696705    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cac73b5-4073-48b9-a4e8-6d12264ca322" containerName="revoke-keys"
Oct 31 03:49:49.696817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.696808    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cac73b5-4073-48b9-a4e8-6d12264ca322" containerName="revoke-keys"
Oct 31 03:49:49.700584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.700568    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.777178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.777142    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcrp\" (UniqueName: \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.777513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.777468    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.777692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.777547    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.801450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.801306    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:49:49.801450 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.801372    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:49:49.806744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.806712    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"]
Oct 31 03:49:49.878302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.878262    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.878497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.878328    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.878497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.878479    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcrp\" (UniqueName: \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.879077 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.879028    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.880905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.880884    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:49.937856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:49.937823    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcrp\" (UniqueName: \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\") pod \"maas-api-revoke-sub-92315a5a-lnplc\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:50.023608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:50.015290    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:50.422003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:50.421975    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"]
Oct 31 03:49:50.423848 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:50.423818    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b71738_401f_428e_b569_29f5f5fbedcf.slice/crio-b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72 WatchSource:0}: Error finding container b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72: Status 404 returned error can't find the container with id b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72
Oct 31 03:49:51.235778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:51.235741    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc" event={"ID":"b6b71738-401f-428e-b569-29f5f5fbedcf","Type":"ContainerStarted","Data":"34c53307d528dca1b15c56b7ad56fda7d474408b7ff9695a27a90e675fa4a6f7"}
Oct 31 03:49:51.236191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:51.235828    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc" event={"ID":"b6b71738-401f-428e-b569-29f5f5fbedcf","Type":"ContainerStarted","Data":"b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72"}
Oct 31 03:49:51.342914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:51.342858    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc" podStartSLOduration=2.342843555 podStartE2EDuration="2.342843555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:51.341482735 +0000 UTC m=+2541.356044445" watchObservedRunningTime="2026-10-31 03:49:51.342843555 +0000 UTC m=+2541.357405265"
Oct 31 03:49:52.244491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:52.244453    5831 generic.go:356] "Generic (PLEG): container finished" podID="b6b71738-401f-428e-b569-29f5f5fbedcf" containerID="34c53307d528dca1b15c56b7ad56fda7d474408b7ff9695a27a90e675fa4a6f7" exitCode=0
Oct 31 03:49:52.244960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:52.244575    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc" event={"ID":"b6b71738-401f-428e-b569-29f5f5fbedcf","Type":"ContainerDied","Data":"34c53307d528dca1b15c56b7ad56fda7d474408b7ff9695a27a90e675fa4a6f7"}
Oct 31 03:49:52.978356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:52.978290    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:49:53.016099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016065    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016263    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016302    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016336    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016365    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016470    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016569    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.016604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016584    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:53.016604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.016598    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.017070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017017    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:53.017070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017065    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:53.017364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017336    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.017584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017370    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.017584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017387    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c4e804-4847-4652-b2a6-23f68c2fe550-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.017584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.017484    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:53.019273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.019248    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:53.022964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.019505    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:53.022964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.019629    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:49:53.022964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.019804    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "kube-api-access-q7hct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:53.117989 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.117952    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\") pod \"24c4e804-4847-4652-b2a6-23f68c2fe550\" (UID: \"24c4e804-4847-4652-b2a6-23f68c2fe550\") "
Oct 31 03:49:53.118277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118253    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket" pod "24c4e804-4847-4652-b2a6-23f68c2fe550" (UID: "24c4e804-4847-4652-b2a6-23f68c2fe550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118489    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118507    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118516    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118540    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c4e804-4847-4652-b2a6-23f68c2fe550-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118550    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hct\" (UniqueName: \"kubernetes.io/projected/24c4e804-4847-4652-b2a6-23f68c2fe550-kube-api-access-q7hct\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.118562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.118560    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c4e804-4847-4652-b2a6-23f68c2fe550-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.254213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.254119    5831 generic.go:356] "Generic (PLEG): container finished" podID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerID="0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a" exitCode=0
Oct 31 03:49:53.254213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.254177    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"
Oct 31 03:49:53.254789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.254214    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerDied","Data":"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"}
Oct 31 03:49:53.254789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.254288    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq" event={"ID":"24c4e804-4847-4652-b2a6-23f68c2fe550","Type":"ContainerDied","Data":"8961dd480cb2b1a83d224cc2745b4f0e1d7c4490cd970ac6f4ef760375d31674"}
Oct 31 03:49:53.254789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.254322    5831 scope.go:122] "RemoveContainer" containerID="0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"
Oct 31 03:49:53.267449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.267430    5831 scope.go:122] "RemoveContainer" containerID="efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"
Oct 31 03:49:53.280420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.280394    5831 scope.go:122] "RemoveContainer" containerID="0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"
Oct 31 03:49:53.280793 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:53.280773    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a\": container with ID starting with 0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a not found: ID does not exist" containerID="0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"
Oct 31 03:49:53.280911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.280801    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a"} err="failed to get container status \"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a\": rpc error: code = NotFound desc = could not find container \"0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a\": container with ID starting with 0ed35cf6520f9fbb7dec0ed13044e0fffa237f8ad86f56115b16dd51ca81863a not found: ID does not exist"
Oct 31 03:49:53.280911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.280820    5831 scope.go:122] "RemoveContainer" containerID="efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"
Oct 31 03:49:53.281081 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:49:53.281061    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781\": container with ID starting with efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781 not found: ID does not exist" containerID="efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"
Oct 31 03:49:53.281158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.281090    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781"} err="failed to get container status \"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781\": rpc error: code = NotFound desc = could not find container \"efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781\": container with ID starting with efc1f3fe9f6964a8e10f36a0b3d0fb92d4087f816652c40514ed361d66052781 not found: ID does not exist"
Oct 31 03:49:53.409492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.409460    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:49:53.415024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.414996    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc85c7-openshift-default-7bdf484f7-zjhcq"]
Oct 31 03:49:53.415233 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.415210    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:53.422949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.422922    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\") pod \"b6b71738-401f-428e-b569-29f5f5fbedcf\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") "
Oct 31 03:49:53.423053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.422973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\" (UniqueName: \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\") pod \"b6b71738-401f-428e-b569-29f5f5fbedcf\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") "
Oct 31 03:49:53.423132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.423065    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\") pod \"b6b71738-401f-428e-b569-29f5f5fbedcf\" (UID: \"b6b71738-401f-428e-b569-29f5f5fbedcf\") "
Oct 31 03:49:53.423420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.423391    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca" pod "b6b71738-401f-428e-b569-29f5f5fbedcf" (UID: "b6b71738-401f-428e-b569-29f5f5fbedcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:53.423662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.423610    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.425294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.425265    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca" pod "b6b71738-401f-428e-b569-29f5f5fbedcf" (UID: "b6b71738-401f-428e-b569-29f5f5fbedcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:53.425294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.425278    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp" pod "b6b71738-401f-428e-b569-29f5f5fbedcf" (UID: "b6b71738-401f-428e-b569-29f5f5fbedcf"). InnerVolumeSpecName "kube-api-access-mrcrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:53.525090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.524798    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b71738-401f-428e-b569-29f5f5fbedcf-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.525090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.524835    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcrp\" (UniqueName: \"kubernetes.io/projected/b6b71738-401f-428e-b569-29f5f5fbedcf-kube-api-access-mrcrp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:53.937143 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937111    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:49:53.937687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937673    5831 cpu_manager.go:414] "containerMap: removing container" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:53.937755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937689    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:53.937755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937729    5831 cpu_manager.go:414] "containerMap: removing container" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:53.937755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937735    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:53.937755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937744    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b6b71738-401f-428e-b569-29f5f5fbedcf" containerName="revoke-keys"
Oct 31 03:49:53.937755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937749    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b71738-401f-428e-b569-29f5f5fbedcf" containerName="revoke-keys"
Oct 31 03:49:53.937912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937837    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:53.937912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.937845    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b71738-401f-428e-b569-29f5f5fbedcf" containerName="revoke-keys"
Oct 31 03:49:53.941523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.941508    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:53.993579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.990900    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e2f0bd\""
Oct 31 03:49:53.993579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.991079    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e2f0bd-dockercfg-8g9z5\""
Oct 31 03:49:53.998258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:53.998227    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:49:54.099109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.099076    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:49:54.100221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.100037    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" containerName="istio-proxy"
Oct 31 03:49:54.118957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.118930    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.148500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.148464    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c4e804-4847-4652-b2a6-23f68c2fe550" path="/var/lib/kubelet/pods/24c4e804-4847-4652-b2a6-23f68c2fe550/volumes"
Oct 31 03:49:54.186854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.186691    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:49:54.188695 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.188624    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnr7k\" (UniqueName: \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.189166 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.189055    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.288342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.288297    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc" event={"ID":"b6b71738-401f-428e-b569-29f5f5fbedcf","Type":"ContainerDied","Data":"b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72"}
Oct 31 03:49:54.288823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.288379    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b01a928e9ce456eebefcb91942be48575600514f5aa9299230a15fd678ab2c72"
Oct 31 03:49:54.288823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.288480    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"
Oct 31 03:49:54.291487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.290855    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8txw\" (UniqueName: \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.291487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.290927    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnr7k\" (UniqueName: \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.291487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.291009    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.291487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.291082    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.292485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.292461    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.345137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.345108    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnr7k\" (UniqueName: \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\") pod \"payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.392428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.392387    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8txw\" (UniqueName: \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.392680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.392490    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.393396 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.393364    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.446256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.446179    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8txw\" (UniqueName: \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\") pod \"payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.467697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.467670    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:54.556168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.556138    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:54.851494 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.851465    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:49:54.852152 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:54.852127    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd353fc33_79fb_42e1_a116_34e6fb182555.slice/crio-0d0375b61d6cdc60d78e153769cd7830566bf9fcfc5702a1694894e201599297 WatchSource:0}: Error finding container 0d0375b61d6cdc60d78e153769cd7830566bf9fcfc5702a1694894e201599297: Status 404 returned error can't find the container with id 0d0375b61d6cdc60d78e153769cd7830566bf9fcfc5702a1694894e201599297
Oct 31 03:49:54.955264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:54.955233    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:49:54.956006 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:54.955979    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc92ffe9_dcd1_400f_96ed_13db87829653.slice/crio-e8a2c8b3db97d5f5bfb9c4a81d9e9542fe854d68a6620a9273e9a5bc214f46dc WatchSource:0}: Error finding container e8a2c8b3db97d5f5bfb9c4a81d9e9542fe854d68a6620a9273e9a5bc214f46dc: Status 404 returned error can't find the container with id e8a2c8b3db97d5f5bfb9c4a81d9e9542fe854d68a6620a9273e9a5bc214f46dc
Oct 31 03:49:55.300291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.300254    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" event={"ID":"d353fc33-79fb-42e1-a116-34e6fb182555","Type":"ContainerStarted","Data":"45ede22ad12e129c452b4e02e6cc37360c2840cecfef43d2343da9a60843d0d7"}
Oct 31 03:49:55.300860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.300323    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" event={"ID":"d353fc33-79fb-42e1-a116-34e6fb182555","Type":"ContainerStarted","Data":"0d0375b61d6cdc60d78e153769cd7830566bf9fcfc5702a1694894e201599297"}
Oct 31 03:49:55.300860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.300351    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:49:55.302103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.302076    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" event={"ID":"dc92ffe9-dcd1-400f-96ed-13db87829653","Type":"ContainerStarted","Data":"cee79b5498e1908599d0914a4e50e08f47c829060e13a7e1581ed48cd2323cc2"}
Oct 31 03:49:55.302242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.302120    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" event={"ID":"dc92ffe9-dcd1-400f-96ed-13db87829653","Type":"ContainerStarted","Data":"e8a2c8b3db97d5f5bfb9c4a81d9e9542fe854d68a6620a9273e9a5bc214f46dc"}
Oct 31 03:49:55.302242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.302152    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:49:55.408572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.408498    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" podStartSLOduration=1.4084837559999999 podStartE2EDuration="1.408483756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:55.406301525 +0000 UTC m=+2545.420863233" watchObservedRunningTime="2026-10-31 03:49:55.408483756 +0000 UTC m=+2545.423045465"
Oct 31 03:49:55.513975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:55.513907    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" podStartSLOduration=2.513889551 podStartE2EDuration="2.513889551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:55.511468716 +0000 UTC m=+2545.526030424" watchObservedRunningTime="2026-10-31 03:49:55.513889551 +0000 UTC m=+2545.528451328"
Oct 31 03:49:56.097994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.097951    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:49:56.103425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.103395    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.117756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.117727    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck5rg\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.117887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.117777    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.117887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.117810    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.117887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.117865    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.118012 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.117962    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.118168 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.118145    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.118264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.118245    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.118311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.118289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.118362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.118317    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.191816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.191778    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:49:56.219134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219095    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck5rg\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219154    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219188    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219216    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219244    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219402    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219545 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219507    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219601    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.219685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219629    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.220171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.219923    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.220408 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.220379    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.220497 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.220378    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.220706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.220661    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.220915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.220897    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.223007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.222976    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.224306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.224281    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.272828 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.272804    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck5rg\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.273392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.273372    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.421963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.421871    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:56.851284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.851255    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:49:56.852896 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:49:56.852867    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bb4852_ad0f_498c_9597_e868ba9556dc.slice/crio-28d385b3984016be8d5f6b5a79f88e9253f37d7a36dacdbb320bfbcf4c7e900f WatchSource:0}: Error finding container 28d385b3984016be8d5f6b5a79f88e9253f37d7a36dacdbb320bfbcf4c7e900f: Status 404 returned error can't find the container with id 28d385b3984016be8d5f6b5a79f88e9253f37d7a36dacdbb320bfbcf4c7e900f
Oct 31 03:49:56.855427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.855392    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:56.855569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.855474    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:56.855569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:56.855520    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:49:57.321385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.321343    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" event={"ID":"e8bb4852-ad0f-498c-9597-e868ba9556dc","Type":"ContainerStarted","Data":"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"}
Oct 31 03:49:57.321600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.321412    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" event={"ID":"e8bb4852-ad0f-498c-9597-e868ba9556dc","Type":"ContainerStarted","Data":"28d385b3984016be8d5f6b5a79f88e9253f37d7a36dacdbb320bfbcf4c7e900f"}
Oct 31 03:49:57.422213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.422174    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:49:57.424099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.424070    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:49:57.424272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.424130    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:49:57.429317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:57.429269    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podStartSLOduration=1.429254153 podStartE2EDuration="1.429254153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:57.428259896 +0000 UTC m=+2547.442821605" watchObservedRunningTime="2026-10-31 03:49:57.429254153 +0000 UTC m=+2547.443815862"
Oct 31 03:49:58.422962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:58.422916    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:49:58.423402 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:58.422983    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:49:59.423209 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.423172    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:49:59.423644 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.423232    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:49:59.429492 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.429463    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:49:59.433936 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.433918    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459618    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459670    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459833 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459779    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9j4\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459867 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459833    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459908 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459889    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.459961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.459946    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.460134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.460119    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.460326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.460307    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.460425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.460411    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.542602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.542514    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:49:59.561793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.561753    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.561996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.561868    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.561996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.561938    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.561996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.561981    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562009    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562041    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562078    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9j4\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562108    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562127    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.562928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.562898    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.563422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.563399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.564234 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.564211    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.564646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.564623    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.568789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.568754    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.571179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.571147    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.625597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.621510    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.634619 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.634584    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.637136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.637108    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9j4\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:49:59.748542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:49:59.748494    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:50:00.207969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.207754    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:50:00.211505 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:00.211478    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613fdee2_7f61_4328_af15_65a992d2351a.slice/crio-27fc79474fa3641d9203e72bc152a0ab2ed627ca007ec78787e765e3008574ef WatchSource:0}: Error finding container 27fc79474fa3641d9203e72bc152a0ab2ed627ca007ec78787e765e3008574ef: Status 404 returned error can't find the container with id 27fc79474fa3641d9203e72bc152a0ab2ed627ca007ec78787e765e3008574ef
Oct 31 03:50:00.214057 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.214028    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:00.214131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.214096    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:00.214131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.214124    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:00.362909 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.362868    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" event={"ID":"613fdee2-7f61-4328-af15-65a992d2351a","Type":"ContainerStarted","Data":"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"}
Oct 31 03:50:00.363096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.362941    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" event={"ID":"613fdee2-7f61-4328-af15-65a992d2351a","Type":"ContainerStarted","Data":"27fc79474fa3641d9203e72bc152a0ab2ed627ca007ec78787e765e3008574ef"}
Oct 31 03:50:00.415295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.415254    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"]
Oct 31 03:50:00.419698 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.419671    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.422272 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.422244    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:50:00.422419 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.422297    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:50:00.461649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.461573    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"]
Oct 31 03:50:00.471920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.470858    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:00.471920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.470874    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:00.471920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.471757    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.471920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.471913    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.472997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.471983    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ljn\" (UniqueName: \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.475559 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.475484    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podStartSLOduration=1.475466787 podStartE2EDuration="1.475466787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:00.470998671 +0000 UTC m=+2550.485560380" watchObservedRunningTime="2026-10-31 03:50:00.475466787 +0000 UTC m=+2550.490028496"
Oct 31 03:50:00.572713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.572668    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.572922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.572750    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ljn\" (UniqueName: \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.572922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.572842    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.573593 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.573571    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.575581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.575554    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.625777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.625745    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ljn\" (UniqueName: \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\") pod \"maas-api-revoke-sub-0229bc26-l4qq4\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.735016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.734928    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:00.753562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:00.753504    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:50:01.150386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.149768    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"]
Oct 31 03:50:01.150386 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:01.150237    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886bd70c_8d32_4614_96e4_ee3ab3592bcc.slice/crio-b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3 WatchSource:0}: Error finding container b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3: Status 404 returned error can't find the container with id b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3
Oct 31 03:50:01.423303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.423200    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:50:01.423598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.423562    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:50:01.424249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.424215    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4" event={"ID":"886bd70c-8d32-4614-96e4-ee3ab3592bcc","Type":"ContainerStarted","Data":"b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3"}
Oct 31 03:50:01.752069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.752037    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:01.752414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:01.752096    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:02.434613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.432493    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:50:02.434613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.432612    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:50:02.445220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.445187    5831 generic.go:356] "Generic (PLEG): container finished" podID="886bd70c-8d32-4614-96e4-ee3ab3592bcc" containerID="d2bf2b48be01c506d6dbb9815863dc3ec1275489c0115209e7bf1f1ff5d8a34c" exitCode=0
Oct 31 03:50:02.445418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.445290    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4" event={"ID":"886bd70c-8d32-4614-96e4-ee3ab3592bcc","Type":"ContainerDied","Data":"d2bf2b48be01c506d6dbb9815863dc3ec1275489c0115209e7bf1f1ff5d8a34c"}
Oct 31 03:50:02.749793 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.749761    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:02.749968 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:02.749826    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:03.423132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.423092    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:50:03.423583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.423169    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:50:03.712554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.711962    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:03.749336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.749299    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:03.749536 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.749353    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:03.775881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.775850    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\") pod \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") "
Oct 31 03:50:03.776081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.775979    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\" (UniqueName: \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\") pod \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") "
Oct 31 03:50:03.776151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.776093    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\") pod \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\" (UID: \"886bd70c-8d32-4614-96e4-ee3ab3592bcc\") "
Oct 31 03:50:03.776330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.776303    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca" pod "886bd70c-8d32-4614-96e4-ee3ab3592bcc" (UID: "886bd70c-8d32-4614-96e4-ee3ab3592bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:03.776622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.776597    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:03.778386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.778358    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca" pod "886bd70c-8d32-4614-96e4-ee3ab3592bcc" (UID: "886bd70c-8d32-4614-96e4-ee3ab3592bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:03.815589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.806745    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn" pod "886bd70c-8d32-4614-96e4-ee3ab3592bcc" (UID: "886bd70c-8d32-4614-96e4-ee3ab3592bcc"). InnerVolumeSpecName "kube-api-access-c6ljn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:03.877427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.877390    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ljn\" (UniqueName: \"kubernetes.io/projected/886bd70c-8d32-4614-96e4-ee3ab3592bcc-kube-api-access-c6ljn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:03.877427 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:03.877421    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886bd70c-8d32-4614-96e4-ee3ab3592bcc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.466940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.466907    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"
Oct 31 03:50:04.466940 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.466924    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4" event={"ID":"886bd70c-8d32-4614-96e4-ee3ab3592bcc","Type":"ContainerDied","Data":"b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3"}
Oct 31 03:50:04.467400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.466981    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73a410d8a6c1bb2d8bfbe8676f0795ae6fae4160a07cf63bb72b162dd185bb3"
Oct 31 03:50:04.749177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.749093    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:04.749177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.749153    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:04.810556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.810048    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:50:04.811132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.811103    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:50:04.812356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:04.812333    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:50:05.151083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.150987    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"]
Oct 31 03:50:05.151774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.151751    5831 cpu_manager.go:414] "containerMap: removing container" podUID="886bd70c-8d32-4614-96e4-ee3ab3592bcc" containerName="revoke-keys"
Oct 31 03:50:05.151774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.151774    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886bd70c-8d32-4614-96e4-ee3ab3592bcc" containerName="revoke-keys"
Oct 31 03:50:05.151969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.151941    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="886bd70c-8d32-4614-96e4-ee3ab3592bcc" containerName="revoke-keys"
Oct 31 03:50:05.156137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.156115    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.218038 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.217905    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:05.218734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.218709    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:05.224910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.224795    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"]
Oct 31 03:50:05.351009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.350971    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5nn\" (UniqueName: \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.351195 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.351028    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.351241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.351188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.452357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.452320    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.452611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.452467    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5nn\" (UniqueName: \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.452611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.452566    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.453124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.453106    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.455221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.455198    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.513376 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.513336    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5nn\" (UniqueName: \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\") pod \"maas-api-revoke-sub-0f3b76e9-kmpst\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:05.749179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.749093    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:05.749179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.749158    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:05.771686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:05.771645    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:06.168435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.168405    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"]
Oct 31 03:50:06.169169 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:06.169145    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa54cc50_4f91_4262_8bc4_5fdf0cda9eb9.slice/crio-e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02 WatchSource:0}: Error finding container e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02: Status 404 returned error can't find the container with id e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02
Oct 31 03:50:06.313598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.313569    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:50:06.313783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.313638    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:50:06.491613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.491572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst" event={"ID":"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9","Type":"ContainerStarted","Data":"e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02"}
Oct 31 03:50:06.749988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.749889    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 31 03:50:06.749988 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:06.749955    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 31 03:50:07.504041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:07.503999    5831 generic.go:356] "Generic (PLEG): container finished" podID="aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" containerID="27d1426470b3162e284599ed0d28e64a372981e953eb6b70e5df3ad7ced976e4" exitCode=0
Oct 31 03:50:07.504224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:07.504090    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst" event={"ID":"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9","Type":"ContainerDied","Data":"27d1426470b3162e284599ed0d28e64a372981e953eb6b70e5df3ad7ced976e4"}
Oct 31 03:50:07.753627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:07.753598    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:50:07.754140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:07.754092    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:50:07.754865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:07.754846    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:50:08.677439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.677408    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:08.696694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.696662    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\") pod \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") "
Oct 31 03:50:08.696886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.696719    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\") pod \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") "
Oct 31 03:50:08.696886 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.696876    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\" (UniqueName: \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\") pod \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\" (UID: \"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9\") "
Oct 31 03:50:08.697128 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.697103    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca" pod "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" (UID: "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:08.697540 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.697471    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:08.698929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.698905    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca" pod "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" (UID: "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:08.699257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.699238    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn" pod "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" (UID: "aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9"). InnerVolumeSpecName "kube-api-access-qj5nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:08.802208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.802112    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:08.802773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:08.802733    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5nn\" (UniqueName: \"kubernetes.io/projected/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9-kube-api-access-qj5nn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:09.523761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:09.523732    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"
Oct 31 03:50:09.524108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:09.523772    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst" event={"ID":"aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9","Type":"ContainerDied","Data":"e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02"}
Oct 31 03:50:09.524108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:09.523832    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62ff95aa8d19d25cfed363ab7633bcf5f118afae09514d53308c59ab18fbb02"
Oct 31 03:50:10.998444 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:10.998400    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"]
Oct 31 03:50:10.999478 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:10.999455    5831 cpu_manager.go:414] "containerMap: removing container" podUID="aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" containerName="revoke-keys"
Oct 31 03:50:10.999546 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:10.999483    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" containerName="revoke-keys"
Oct 31 03:50:10.999686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:10.999673    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" containerName="revoke-keys"
Oct 31 03:50:11.004252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.004232    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.040196 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.040167    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.040359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.040319    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.040417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.040397    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdd7\" (UniqueName: \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.054183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.054150    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"]
Oct 31 03:50:11.094081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.094048    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:11.094285 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.094211    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:11.141591 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.141558    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdd7\" (UniqueName: \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.141829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.141684    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.141829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.141817    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.142575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.142547    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.144411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.144388    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.194831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.194803    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdd7\" (UniqueName: \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\") pod \"maas-api-revoke-sub-7e79be76-bh9wr\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.320957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.320874    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:11.702643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:11.702610    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"]
Oct 31 03:50:11.704139 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:11.704103    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc124b6d3_9b28_4230_ba43_3f60ab4eb215.slice/crio-e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b WatchSource:0}: Error finding container e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b: Status 404 returned error can't find the container with id e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b
Oct 31 03:50:12.549933 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:12.549894    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr" event={"ID":"c124b6d3-9b28-4230-ba43-3f60ab4eb215","Type":"ContainerStarted","Data":"43a2fd91420744de9f5174c22f861172004efd79d789e5dc5406ad0cb2ec554f"}
Oct 31 03:50:12.550416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:12.549950    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr" event={"ID":"c124b6d3-9b28-4230-ba43-3f60ab4eb215","Type":"ContainerStarted","Data":"e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b"}
Oct 31 03:50:12.654926 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:12.654879    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr" podStartSLOduration=2.654864426 podStartE2EDuration="2.654864426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:12.653706696 +0000 UTC m=+2562.668268403" watchObservedRunningTime="2026-10-31 03:50:12.654864426 +0000 UTC m=+2562.669426134"
Oct 31 03:50:12.979137 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:50:12.979083    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:50:13.559155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:13.559124    5831 generic.go:356] "Generic (PLEG): container finished" podID="c124b6d3-9b28-4230-ba43-3f60ab4eb215" containerID="43a2fd91420744de9f5174c22f861172004efd79d789e5dc5406ad0cb2ec554f" exitCode=0
Oct 31 03:50:13.559604 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:13.559219    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr" event={"ID":"c124b6d3-9b28-4230-ba43-3f60ab4eb215","Type":"ContainerDied","Data":"43a2fd91420744de9f5174c22f861172004efd79d789e5dc5406ad0cb2ec554f"}
Oct 31 03:50:14.199064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.199030    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:50:14.739993 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.739964    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:14.799742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.799711    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\" (UniqueName: \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\") pod \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") "
Oct 31 03:50:14.799920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.799823    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\") pod \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") "
Oct 31 03:50:14.799920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.799852    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\") pod \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\" (UID: \"c124b6d3-9b28-4230-ba43-3f60ab4eb215\") "
Oct 31 03:50:14.800301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.800270    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca" pod "c124b6d3-9b28-4230-ba43-3f60ab4eb215" (UID: "c124b6d3-9b28-4230-ba43-3f60ab4eb215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:14.802029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.801997    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7" pod "c124b6d3-9b28-4230-ba43-3f60ab4eb215" (UID: "c124b6d3-9b28-4230-ba43-3f60ab4eb215"). InnerVolumeSpecName "kube-api-access-chdd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:14.802149 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.802129    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca" pod "c124b6d3-9b28-4230-ba43-3f60ab4eb215" (UID: "c124b6d3-9b28-4230-ba43-3f60ab4eb215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:14.901964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.901871    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdd7\" (UniqueName: \"kubernetes.io/projected/c124b6d3-9b28-4230-ba43-3f60ab4eb215-kube-api-access-chdd7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:14.902201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.902181    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:14.902308 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:14.902296    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c124b6d3-9b28-4230-ba43-3f60ab4eb215-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:15.628556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:15.622454    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr" event={"ID":"c124b6d3-9b28-4230-ba43-3f60ab4eb215","Type":"ContainerDied","Data":"e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b"}
Oct 31 03:50:15.628556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:15.622523    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53e2c75cff0b87cfedb8d99d4918c83b2d2c3b9e0c20205714c09fc35c0b11b"
Oct 31 03:50:15.628556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:15.622622    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"
Oct 31 03:50:17.174557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.174492    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"]
Oct 31 03:50:17.175121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.175105    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c124b6d3-9b28-4230-ba43-3f60ab4eb215" containerName="revoke-keys"
Oct 31 03:50:17.175182 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.175123    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c124b6d3-9b28-4230-ba43-3f60ab4eb215" containerName="revoke-keys"
Oct 31 03:50:17.175259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.175250    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c124b6d3-9b28-4230-ba43-3f60ab4eb215" containerName="revoke-keys"
Oct 31 03:50:17.178435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.178414    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.233388 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.233347    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"]
Oct 31 03:50:17.272509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.272476    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:17.272706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.272541    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:17.277866 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.277841    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.278011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.277973    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbwc\" (UniqueName: \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.278075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.278057    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.379623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.379580    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbwc\" (UniqueName: \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.379836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.379674    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.379836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.379779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.380390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.380362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.382426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.382395    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.435100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.435071    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbwc\" (UniqueName: \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\") pod \"maas-api-revoke-sub-97c7b27b-gbf2r\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.494567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.494515    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:17.867303 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:17.867268    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"]
Oct 31 03:50:17.868071 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:17.868042    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5072f4c6_0316_485a_80fd_ba2d97f08dcc.slice/crio-02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22 WatchSource:0}: Error finding container 02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22: Status 404 returned error can't find the container with id 02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22
Oct 31 03:50:18.646868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:18.646703    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r" event={"ID":"5072f4c6-0316-485a-80fd-ba2d97f08dcc","Type":"ContainerStarted","Data":"9eeffcb6be384c8e5b004e2f57f8c57317709c921307aa3509e1bbc4f603febc"}
Oct 31 03:50:18.646868 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:18.646774    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r" event={"ID":"5072f4c6-0316-485a-80fd-ba2d97f08dcc","Type":"ContainerStarted","Data":"02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22"}
Oct 31 03:50:18.752859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:18.752808    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r" podStartSLOduration=1.7527932019999999 podStartE2EDuration="1.752793202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:18.752095685 +0000 UTC m=+2568.766657392" watchObservedRunningTime="2026-10-31 03:50:18.752793202 +0000 UTC m=+2568.767354911"
Oct 31 03:50:19.657277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:19.657235    5831 generic.go:356] "Generic (PLEG): container finished" podID="5072f4c6-0316-485a-80fd-ba2d97f08dcc" containerID="9eeffcb6be384c8e5b004e2f57f8c57317709c921307aa3509e1bbc4f603febc" exitCode=0
Oct 31 03:50:19.658017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:19.657346    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r" event={"ID":"5072f4c6-0316-485a-80fd-ba2d97f08dcc","Type":"ContainerDied","Data":"9eeffcb6be384c8e5b004e2f57f8c57317709c921307aa3509e1bbc4f603febc"}
Oct 31 03:50:20.833367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.833339    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:20.927009 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.926925    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\") pod \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") "
Oct 31 03:50:20.927199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.927039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\" (UniqueName: \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\") pod \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") "
Oct 31 03:50:20.927199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.927067    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\") pod \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\" (UID: \"5072f4c6-0316-485a-80fd-ba2d97f08dcc\") "
Oct 31 03:50:20.927467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.927444    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca" pod "5072f4c6-0316-485a-80fd-ba2d97f08dcc" (UID: "5072f4c6-0316-485a-80fd-ba2d97f08dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:20.929334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.929302    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca" pod "5072f4c6-0316-485a-80fd-ba2d97f08dcc" (UID: "5072f4c6-0316-485a-80fd-ba2d97f08dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:20.929447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:20.929348    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc" pod "5072f4c6-0316-485a-80fd-ba2d97f08dcc" (UID: "5072f4c6-0316-485a-80fd-ba2d97f08dcc"). InnerVolumeSpecName "kube-api-access-xnbwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:21.028275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.028240    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:21.028275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.028271    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbwc\" (UniqueName: \"kubernetes.io/projected/5072f4c6-0316-485a-80fd-ba2d97f08dcc-kube-api-access-xnbwc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:21.028275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.028281    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5072f4c6-0316-485a-80fd-ba2d97f08dcc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:21.676113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.676080    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"
Oct 31 03:50:21.676326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.676117    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r" event={"ID":"5072f4c6-0316-485a-80fd-ba2d97f08dcc","Type":"ContainerDied","Data":"02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22"}
Oct 31 03:50:21.676326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.676175    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c807dfb7bddf7484f7ca50786ce0cf39b10d6bfc6ee2d34a70c9d27778af22"
Oct 31 03:50:21.786853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.786822    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"]
Oct 31 03:50:21.787510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.787491    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5072f4c6-0316-485a-80fd-ba2d97f08dcc" containerName="revoke-keys"
Oct 31 03:50:21.787510 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.787509    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5072f4c6-0316-485a-80fd-ba2d97f08dcc" containerName="revoke-keys"
Oct 31 03:50:21.787717 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.787663    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5072f4c6-0316-485a-80fd-ba2d97f08dcc" containerName="revoke-keys"
Oct 31 03:50:21.791669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.791649    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:21.815928 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.815899    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"]
Oct 31 03:50:21.839848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.839818    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:21.840346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.839916    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:21.939460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.939423    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:21.939684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.939620    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg52\" (UniqueName: \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:21.939735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:21.939689    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.040406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.040367    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg52\" (UniqueName: \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.040406 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.040401    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.040716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.040494    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.041354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.041324    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.043084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.043063    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.093920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.093889    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg52\" (UniqueName: \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\") pod \"maas-api-revoke-sub-893e053b-mzlpq\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.108700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.108674    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:22.491582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.491547    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"]
Oct 31 03:50:22.492445 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:22.492421    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ecec33_e33a_40fb_a818_ec5368293613.slice/crio-267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62 WatchSource:0}: Error finding container 267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62: Status 404 returned error can't find the container with id 267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62
Oct 31 03:50:22.687297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:22.687265    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq" event={"ID":"71ecec33-e33a-40fb-a818-ec5368293613","Type":"ContainerStarted","Data":"267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62"}
Oct 31 03:50:23.697981 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:23.697897    5831 generic.go:356] "Generic (PLEG): container finished" podID="71ecec33-e33a-40fb-a818-ec5368293613" containerID="42b9cc97e38f4a94c0e22166897ca3cfd8c1df65c6ca646895d91714bf76ef10" exitCode=0
Oct 31 03:50:23.698400 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:23.698026    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq" event={"ID":"71ecec33-e33a-40fb-a818-ec5368293613","Type":"ContainerDied","Data":"42b9cc97e38f4a94c0e22166897ca3cfd8c1df65c6ca646895d91714bf76ef10"}
Oct 31 03:50:24.900000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.899971    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:24.976063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.975986    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\") pod \"71ecec33-e33a-40fb-a818-ec5368293613\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") "
Oct 31 03:50:24.976063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.976055    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\") pod \"71ecec33-e33a-40fb-a818-ec5368293613\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") "
Oct 31 03:50:24.976281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.976167    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\" (UniqueName: \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\") pod \"71ecec33-e33a-40fb-a818-ec5368293613\" (UID: \"71ecec33-e33a-40fb-a818-ec5368293613\") "
Oct 31 03:50:24.976416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.976392    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca" pod "71ecec33-e33a-40fb-a818-ec5368293613" (UID: "71ecec33-e33a-40fb-a818-ec5368293613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:24.976740 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.976678    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:24.978296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.978266    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca" pod "71ecec33-e33a-40fb-a818-ec5368293613" (UID: "71ecec33-e33a-40fb-a818-ec5368293613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:24.978436 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:24.978417    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52" pod "71ecec33-e33a-40fb-a818-ec5368293613" (UID: "71ecec33-e33a-40fb-a818-ec5368293613"). InnerVolumeSpecName "kube-api-access-vzg52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:25.077631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:25.077600    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ecec33-e33a-40fb-a818-ec5368293613-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:25.077631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:25.077629    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzg52\" (UniqueName: \"kubernetes.io/projected/71ecec33-e33a-40fb-a818-ec5368293613-kube-api-access-vzg52\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:25.716543 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:25.716510    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"
Oct 31 03:50:25.716732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:25.716574    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq" event={"ID":"71ecec33-e33a-40fb-a818-ec5368293613","Type":"ContainerDied","Data":"267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62"}
Oct 31 03:50:25.716732 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:25.716622    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267e5ce19520d54d4d8f5918989ea65243b0e32805bb34ff8df78196e2f7bb62"
Oct 31 03:50:26.861349 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.861311    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 03:50:26.862054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.862036    5831 cpu_manager.go:414] "containerMap: removing container" podUID="71ecec33-e33a-40fb-a818-ec5368293613" containerName="revoke-keys"
Oct 31 03:50:26.862119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.862055    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ecec33-e33a-40fb-a818-ec5368293613" containerName="revoke-keys"
Oct 31 03:50:26.862163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.862150    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ecec33-e33a-40fb-a818-ec5368293613" containerName="revoke-keys"
Oct 31 03:50:26.866040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.866021    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:26.914407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.914378    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2339ed-openshift-default-dockercfg-q2fqj\""
Oct 31 03:50:26.967596 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:26.967562    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 03:50:27.010227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010228    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010251    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010385    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010430    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010447    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010545    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6xg\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010659    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.010748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.010703    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111566    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111645 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111673    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6xg\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111770    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111851 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111835    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111890    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.111957 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111928    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.111963    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112002    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112028 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112009    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112042    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112424    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112622    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.112914 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.112889    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.114767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.114742    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.114962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.114943    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.165125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.165097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.165288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.165149    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6xg\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.180111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.180083    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:27.654503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.654467    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 03:50:27.686325 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:27.686294    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d765aa_a51a_4268_b6a1_5a9f6cc64ee8.slice/crio-502af64ab73ae3a628b27df4d024ffe53345717ea554e3fbbe765b930586a82f WatchSource:0}: Error finding container 502af64ab73ae3a628b27df4d024ffe53345717ea554e3fbbe765b930586a82f: Status 404 returned error can't find the container with id 502af64ab73ae3a628b27df4d024ffe53345717ea554e3fbbe765b930586a82f
Oct 31 03:50:27.689197 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.689166    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:27.689296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.689233    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:27.689296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.689261    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:27.750498 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:27.750467    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerStarted","Data":"502af64ab73ae3a628b27df4d024ffe53345717ea554e3fbbe765b930586a82f"}
Oct 31 03:50:28.759856 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:28.759801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerStarted","Data":"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"}
Oct 31 03:50:28.961516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:28.961457    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podStartSLOduration=2.9614405919999998 podStartE2EDuration="2.961440592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:28.913326006 +0000 UTC m=+2578.927887714" watchObservedRunningTime="2026-10-31 03:50:28.961440592 +0000 UTC m=+2578.976002299"
Oct 31 03:50:29.180468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.180435    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:29.186743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.186717    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:29.730294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.730262    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:50:29.752213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.752164    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.832854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.828771    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 03:50:29.838425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.838394    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:50:29.850976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.850950    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:29.852770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.852746    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:50:29.921703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.921672    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.921924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.921741    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.921924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.921843    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922025 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.921977    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.922064    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmw8\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.922206    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.922242    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.922472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:29.922599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:29.922581    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024157    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmw8\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024216    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024236    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024261    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024325    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024384    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024440    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024495    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024567 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024553    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.024954 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.024933    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.025075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.025047    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.025853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.025425    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.025853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.025835    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.026049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.025838    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.028446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.028417    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.028617 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.028596    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.078023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.077990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmw8\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.078184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.077996    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.135113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.135079    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:30.564921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.564893    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:50:30.565599 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:30.565523    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58de3d62_c934_41d0_91c6_7c9b18129df0.slice/crio-51285f7312950c02df0a6454e64a6c297371ec7470f6bf2c4fe97f1ba2d47eb1 WatchSource:0}: Error finding container 51285f7312950c02df0a6454e64a6c297371ec7470f6bf2c4fe97f1ba2d47eb1: Status 404 returned error can't find the container with id 51285f7312950c02df0a6454e64a6c297371ec7470f6bf2c4fe97f1ba2d47eb1
Oct 31 03:50:30.568385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.568358    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:30.568469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.568422    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:30.568469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.568451    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:30.861820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.861723    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" event={"ID":"58de3d62-c934-41d0-91c6-7c9b18129df0","Type":"ContainerStarted","Data":"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"}
Oct 31 03:50:30.861820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.861793    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" event={"ID":"58de3d62-c934-41d0-91c6-7c9b18129df0","Type":"ContainerStarted","Data":"51285f7312950c02df0a6454e64a6c297371ec7470f6bf2c4fe97f1ba2d47eb1"}
Oct 31 03:50:30.972832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:30.972772    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podStartSLOduration=1.972757408 podStartE2EDuration="1.972757408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:30.969759777 +0000 UTC m=+2580.984321486" watchObservedRunningTime="2026-10-31 03:50:30.972757408 +0000 UTC m=+2580.987319115"
Oct 31 03:50:31.135511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:31.135433    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:31.136929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:31.136904    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:31.137069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:31.136949    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:32.135472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:32.135434    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:32.136035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:32.135501    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:33.154552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:33.142726    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:33.154552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:33.142785    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:34.151558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:34.148937    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:34.151558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:34.149002    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:35.148271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:35.148235    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:35.149712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:35.149681    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:36.145563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.140937    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:36.183738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.183679    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:36.454935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.454873    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 03:50:36.464372 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.464341    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 03:50:36.464555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.464450    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.516920 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.516880    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2339ed\""
Oct 31 03:50:36.517260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.517243    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2339ed-dockercfg-nq89p\""
Oct 31 03:50:36.601381 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.601334    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshr6\" (UniqueName: \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.601806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.601784    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.708599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.707364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshr6\" (UniqueName: \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.708599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.707460    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.708904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.708882    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.761889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.761857    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshr6\" (UniqueName: \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\") pod \"payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:36.799555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:36.798751    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:37.136018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:37.135933    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:37.136018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:37.135985    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:37.216417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:37.215738    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 03:50:37.275719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:37.275688    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" event={"ID":"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872","Type":"ContainerStarted","Data":"659d87a723180c0f81d3c93f9b59ab1c2c5a363c835eeaddfbfb3ddf4a502e01"}
Oct 31 03:50:38.136398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:38.136363    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:50:38.136597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:38.136413    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:50:38.287563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:38.287474    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" event={"ID":"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872","Type":"ContainerStarted","Data":"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"}
Oct 31 03:50:38.288068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:38.287893    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:38.394575 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:38.394416    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" podStartSLOduration=2.39439687 podStartE2EDuration="2.39439687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:38.393455905 +0000 UTC m=+2588.408017617" watchObservedRunningTime="2026-10-31 03:50:38.39439687 +0000 UTC m=+2588.408958580"
Oct 31 03:50:39.700860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.700818    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"]
Oct 31 03:50:39.706583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.706519    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.758952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.758920    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"]
Oct 31 03:50:39.769157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.769127    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.769313 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.769188    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxk9\" (UniqueName: \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.769375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.769352    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.809391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.809268    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:39.809982 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.809849    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:50:39.870542 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.870477    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.870854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.870733    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.870854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.870807    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxk9\" (UniqueName: \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.872254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.872224    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.873736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.873716    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:39.926071 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:39.926034    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxk9\" (UniqueName: \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\") pod \"maas-api-revoke-sub-ba679f60-qjhv2\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:40.028441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.028340    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:40.179704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.179668    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:50:40.179898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.179727    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:50:40.447998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.447969    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"]
Oct 31 03:50:40.449071 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:40.449041    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f3676d_7652_4968_a3cd_aed5de7f022f.slice/crio-26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662 WatchSource:0}: Error finding container 26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662: Status 404 returned error can't find the container with id 26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662
Oct 31 03:50:40.457505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.456270    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:40.457505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.457451    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:40.459151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:40.459127    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:50:41.354708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.354666    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2" event={"ID":"06f3676d-7652-4968-a3cd-aed5de7f022f","Type":"ContainerStarted","Data":"17448b9886d9720af5faba6f41974f928ac07c510564451f81bcef14c606b105"}
Oct 31 03:50:41.355283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.354732    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2" event={"ID":"06f3676d-7652-4968-a3cd-aed5de7f022f","Type":"ContainerStarted","Data":"26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662"}
Oct 31 03:50:41.434273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.434241    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:50:41.438831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.438813    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.460626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.460567    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2" podStartSLOduration=2.460551287 podStartE2EDuration="2.460551287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:41.459241571 +0000 UTC m=+2591.473803278" watchObservedRunningTime="2026-10-31 03:50:41.460551287 +0000 UTC m=+2591.475112997"
Oct 31 03:50:41.499211 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.499173    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:50:41.572809 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.572771    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.572970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.572836    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.572970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.572910    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.572970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.572961    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4p5g\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.573093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.573047    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.573093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.573083    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.573226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.573199    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.573264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.573248    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.573328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.573317    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674277    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674327    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674359    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674382    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674420    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4p5g\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674451    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674478    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.674703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674652    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.675039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674912    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.675039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.674980    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.675039 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.675027    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.675225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.675122    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.675225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.675167    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.677082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.677053    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.677187 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.677177    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.727175 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.727134    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.727358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.727281    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4p5g\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:41.754314 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:41.754285    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:42.205523 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:42.205495    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dbbd82_7fe2_463a_b5af_b2363d65dcd7.slice/crio-e4b230e95a4e039855c76d971347f95d6a5a2e49bfc2c7256020e14a99917863 WatchSource:0}: Error finding container e4b230e95a4e039855c76d971347f95d6a5a2e49bfc2c7256020e14a99917863: Status 404 returned error can't find the container with id e4b230e95a4e039855c76d971347f95d6a5a2e49bfc2c7256020e14a99917863
Oct 31 03:50:42.207576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.207514    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:50:42.208712 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.208676    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:42.208836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.208758    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:42.208836 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.208803    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:42.365778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.365734    5831 generic.go:356] "Generic (PLEG): container finished" podID="06f3676d-7652-4968-a3cd-aed5de7f022f" containerID="17448b9886d9720af5faba6f41974f928ac07c510564451f81bcef14c606b105" exitCode=0
Oct 31 03:50:42.366275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.365828    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2" event={"ID":"06f3676d-7652-4968-a3cd-aed5de7f022f","Type":"ContainerDied","Data":"17448b9886d9720af5faba6f41974f928ac07c510564451f81bcef14c606b105"}
Oct 31 03:50:42.367943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.367914    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" event={"ID":"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7","Type":"ContainerStarted","Data":"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"}
Oct 31 03:50:42.368089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.367968    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" event={"ID":"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7","Type":"ContainerStarted","Data":"e4b230e95a4e039855c76d971347f95d6a5a2e49bfc2c7256020e14a99917863"}
Oct 31 03:50:42.613418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.613304    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podStartSLOduration=1.6132891040000001 podStartE2EDuration="1.613289104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:42.612001292 +0000 UTC m=+2592.626563000" watchObservedRunningTime="2026-10-31 03:50:42.613289104 +0000 UTC m=+2592.627850812"
Oct 31 03:50:42.754975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:42.754942    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:42.990553 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:50:42.988378    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:50:43.608461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.608432    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:43.612068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.612045    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\") pod \"06f3676d-7652-4968-a3cd-aed5de7f022f\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") "
Oct 31 03:50:43.612148 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.612110    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\") pod \"06f3676d-7652-4968-a3cd-aed5de7f022f\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") "
Oct 31 03:50:43.612204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.612181    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\" (UniqueName: \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\") pod \"06f3676d-7652-4968-a3cd-aed5de7f022f\" (UID: \"06f3676d-7652-4968-a3cd-aed5de7f022f\") "
Oct 31 03:50:43.612377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.612354    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca" pod "06f3676d-7652-4968-a3cd-aed5de7f022f" (UID: "06f3676d-7652-4968-a3cd-aed5de7f022f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:43.612579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.612561    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:43.614375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.614352    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca" pod "06f3676d-7652-4968-a3cd-aed5de7f022f" (UID: "06f3676d-7652-4968-a3cd-aed5de7f022f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:43.614553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.614510    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9" pod "06f3676d-7652-4968-a3cd-aed5de7f022f" (UID: "06f3676d-7652-4968-a3cd-aed5de7f022f"). InnerVolumeSpecName "kube-api-access-5jxk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:43.714174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.714129    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f3676d-7652-4968-a3cd-aed5de7f022f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:43.714174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.714167    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxk9\" (UniqueName: \"kubernetes.io/projected/06f3676d-7652-4968-a3cd-aed5de7f022f-kube-api-access-5jxk9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:43.755958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.755911    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:43.756119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:43.755975    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:44.418549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.418445    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"
Oct 31 03:50:44.418549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.418468    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2" event={"ID":"06f3676d-7652-4968-a3cd-aed5de7f022f","Type":"ContainerDied","Data":"26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662"}
Oct 31 03:50:44.418549 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.418543    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b73bcaaa5d1dca287d4ba24afe5379006246f60ac98671f5ace7a6231c5662"
Oct 31 03:50:44.466726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.466695    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:50:44.467332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.467319    5831 cpu_manager.go:414] "containerMap: removing container" podUID="06f3676d-7652-4968-a3cd-aed5de7f022f" containerName="revoke-keys"
Oct 31 03:50:44.467383 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.467335    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f3676d-7652-4968-a3cd-aed5de7f022f" containerName="revoke-keys"
Oct 31 03:50:44.467464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.467454    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f3676d-7652-4968-a3cd-aed5de7f022f" containerName="revoke-keys"
Oct 31 03:50:44.471394 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.471375    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.519830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.519799    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e2f0bd-serving-cert\""
Oct 31 03:50:44.524023 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.523996    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.524210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.524156    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.524284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.524224    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.524344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.524321    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchxq\" (UniqueName: \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.526571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.526514    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:50:44.625440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.625406    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchxq\" (UniqueName: \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.625985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.625516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.625985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.625624    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.625985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.625679    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.625985 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:50:44.625852    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-serving-cert: secret "maas-api-e2e-praxis-b-e2f0bd-serving-cert" not found
Oct 31 03:50:44.625985 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:50:44.625909    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls podName:b7118eab-d6d0-4ab6-8266-18ecffa94848 nodeName:}" failed. No retries permitted until 2026-10-31 03:50:45.125890654 +0000 UTC m=+2595.140452344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls") pod "maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" (UID: "b7118eab-d6d0-4ab6-8266-18ecffa94848") : secret "maas-api-e2e-praxis-b-e2f0bd-serving-cert" not found
Oct 31 03:50:44.626694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.626673    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.629180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.629113    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.706684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.706651    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchxq\" (UniqueName: \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:44.754756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.754725    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:44.754929 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.754779    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:44.815485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.815443    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:50:44.820707 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.820686    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932406    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932477    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932508    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932566    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932596    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932671    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932728    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnct\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932776    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:44.935433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:44.932819    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055629    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055683    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055707    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055732    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055759    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055867    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnct\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055923    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.056606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.055963    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.057087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.056790    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.057087 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.057045    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.057605 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.057280    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.058907 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.057516    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.059399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.059363    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.063583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.061378    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:50:45.063583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.061518    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.063583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.061969    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.116098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.114464    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.117820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.117789    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnct\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.137302 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.137274    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:45.156947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.156918    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:45.159749 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.159723    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:45.412207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.412117    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:45.609024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.608991    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:50:45.635618 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:50:45.635584    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be57f7a_f741_46a7_8101_3e7aac68daee.slice/crio-1da569f46fef12ccfe6e7e146d777dc3a9db55e99f2abfc928ac2efcd861b6ad WatchSource:0}: Error finding container 1da569f46fef12ccfe6e7e146d777dc3a9db55e99f2abfc928ac2efcd861b6ad: Status 404 returned error can't find the container with id 1da569f46fef12ccfe6e7e146d777dc3a9db55e99f2abfc928ac2efcd861b6ad
Oct 31 03:50:45.638297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.638265    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:45.638411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.638334    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:45.638411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.638364    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:50:45.756343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.756317    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:45.756475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.756372    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:45.822251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:45.822217    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:50:46.444332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.444289    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" event={"ID":"7be57f7a-f741-46a7-8101-3e7aac68daee","Type":"ContainerStarted","Data":"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"}
Oct 31 03:50:46.444506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.444357    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" event={"ID":"7be57f7a-f741-46a7-8101-3e7aac68daee","Type":"ContainerStarted","Data":"1da569f46fef12ccfe6e7e146d777dc3a9db55e99f2abfc928ac2efcd861b6ad"}
Oct 31 03:50:46.445748 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.445719    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" event={"ID":"b7118eab-d6d0-4ab6-8266-18ecffa94848","Type":"ContainerStarted","Data":"3aaed51108ff1cf5a4f93e0019b413b9e234f6d5ee5e05f73dfb3b85709332a4"}
Oct 31 03:50:46.558199 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.558148    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podStartSLOduration=2.558130832 podStartE2EDuration="2.558130832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:46.552184591 +0000 UTC m=+2596.566746298" watchObservedRunningTime="2026-10-31 03:50:46.558130832 +0000 UTC m=+2596.572692533"
Oct 31 03:50:46.755062 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.754972    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:46.755566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:46.755517    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:47.141467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.139670    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:47.143555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.142120    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.143555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.142189    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:47.455912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.455862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" event={"ID":"b7118eab-d6d0-4ab6-8266-18ecffa94848","Type":"ContainerStarted","Data":"1658fad784364fcb1f4d2f0185466198da342a7f293c8c1b8870f2e6870f9879"}
Oct 31 03:50:47.566622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.566565    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" podStartSLOduration=2.529274298 podStartE2EDuration="3.566550356s" totalImagesPullingTime="1.037276058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:47.563826364 +0000 UTC m=+2597.578388066" watchObservedRunningTime="2026-10-31 03:50:47.566550356 +0000 UTC m=+2597.581112105"
Oct 31 03:50:47.758603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.758114    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.758603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:47.758176    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:48.139984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:48.139878    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.140306 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:48.140282    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:48.465992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:48.465964    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:48.755093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:48.755014    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.755093 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:48.755075    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Oct 31 03:50:49.138091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.137988    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:49.138091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.138044    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:49.310064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.310029    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 03:50:49.954259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.952720    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:49.954259 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.954200    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:49.955337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:49.955310    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:50:50.147997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:50.143049    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.147997 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:50.143114    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:51.138253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:51.138216    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.138471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:51.138270    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:52.138198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:52.138159    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.138881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:52.138225    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 31 03:50:53.388003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:53.387972    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:53.388804 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:53.388780    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:53.389942 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:53.389903    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:50:54.481963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:54.481931    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:50:59.978626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:50:59.978383    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 03:51:00.011217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.010344    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.033179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.033154    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 03:51:00.105511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.105479    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-861dfd\""
Oct 31 03:51:00.105511 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.105520    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-861dfd-dockercfg-vkfpf\""
Oct 31 03:51:00.161413 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.161368    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.161616 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.161424    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gjd\" (UniqueName: \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.263359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.263270    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.263359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.263311    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gjd\" (UniqueName: \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.264110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.264080    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.316726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.316692    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gjd\" (UniqueName: \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\") pod \"payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.338714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.338687    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:00.726455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:00.726428    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 03:51:00.727034 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:51:00.726962    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c22c6da_058e_45af_945b_519d82ea39bb.slice/crio-82d8519d3c26d25d5101241cd6de9fa2ced6f7bbe7f5e11203dd5f3a2488210a WatchSource:0}: Error finding container 82d8519d3c26d25d5101241cd6de9fa2ced6f7bbe7f5e11203dd5f3a2488210a: Status 404 returned error can't find the container with id 82d8519d3c26d25d5101241cd6de9fa2ced6f7bbe7f5e11203dd5f3a2488210a
Oct 31 03:51:01.004185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:01.004102    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:51:01.772185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:01.770872    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" event={"ID":"3c22c6da-058e-45af-945b-519d82ea39bb","Type":"ContainerStarted","Data":"8f61c1821b1c8a5e830c0683f9429dff71e52df739152d7aaea5dded13bf6e90"}
Oct 31 03:51:01.772185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:01.770947    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" event={"ID":"3c22c6da-058e-45af-945b-519d82ea39bb","Type":"ContainerStarted","Data":"82d8519d3c26d25d5101241cd6de9fa2ced6f7bbe7f5e11203dd5f3a2488210a"}
Oct 31 03:51:01.772185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:01.772116    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:01.892257 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:01.892200    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" podStartSLOduration=2.892184301 podStartE2EDuration="2.892184301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:01.889199066 +0000 UTC m=+2611.903760775" watchObservedRunningTime="2026-10-31 03:51:01.892184301 +0000 UTC m=+2611.906746009"
Oct 31 03:51:12.981520 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:12.981484    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:51:13.909231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:13.909200    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 03:51:33.028331 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.028285    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:51:33.029179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.029131    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" podUID="f9d0b2ed-2381-49d5-8309-02412b2b3824" containerName="payload-processing" containerID="cri-o://13b48afe81143e2a1d65942e413fe469222000eeb1245809a57ecde980a79177" gracePeriod=30
Oct 31 03:51:33.087377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.087346    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:51:33.087597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.087576    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" podUID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" containerName="payload-pre-processing" containerID="cri-o://94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009" gracePeriod=30
Oct 31 03:51:33.163597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.163545    5831 generic.go:356] "Generic (PLEG): container finished" podID="f9d0b2ed-2381-49d5-8309-02412b2b3824" containerID="13b48afe81143e2a1d65942e413fe469222000eeb1245809a57ecde980a79177" exitCode=0
Oct 31 03:51:33.163798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.163601    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" event={"ID":"f9d0b2ed-2381-49d5-8309-02412b2b3824","Type":"ContainerDied","Data":"13b48afe81143e2a1d65942e413fe469222000eeb1245809a57ecde980a79177"}
Oct 31 03:51:33.408216 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.408188    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:51:33.412252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.412224    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:51:33.472150 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472120    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\" (UniqueName: \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\") pod \"f9d0b2ed-2381-49d5-8309-02412b2b3824\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") "
Oct 31 03:51:33.472326 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472219    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\") pod \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") "
Oct 31 03:51:33.472375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472324    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\") pod \"f9d0b2ed-2381-49d5-8309-02412b2b3824\" (UID: \"f9d0b2ed-2381-49d5-8309-02412b2b3824\") "
Oct 31 03:51:33.472440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472375    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\" (UniqueName: \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\") pod \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\" (UID: \"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f\") "
Oct 31 03:51:33.472715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472676    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume" pod "7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" (UID: "7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:33.472853 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472756    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume" pod "f9d0b2ed-2381-49d5-8309-02412b2b3824" (UID: "f9d0b2ed-2381-49d5-8309-02412b2b3824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:33.472935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472914    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.472992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.472940    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d0b2ed-2381-49d5-8309-02412b2b3824-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.475322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.475301    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7" pod "f9d0b2ed-2381-49d5-8309-02412b2b3824" (UID: "f9d0b2ed-2381-49d5-8309-02412b2b3824"). InnerVolumeSpecName "kube-api-access-cf9r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:33.475486 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.475467    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66" pod "7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" (UID: "7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f"). InnerVolumeSpecName "kube-api-access-26t66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:33.620607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.605826    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26t66\" (UniqueName: \"kubernetes.io/projected/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f-kube-api-access-26t66\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:33.620607 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:33.605866    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9r7\" (UniqueName: \"kubernetes.io/projected/f9d0b2ed-2381-49d5-8309-02412b2b3824-kube-api-access-cf9r7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:34.175622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.175578    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz" event={"ID":"f9d0b2ed-2381-49d5-8309-02412b2b3824","Type":"ContainerDied","Data":"970a5e2d9540ec3f8fca9760ca5b30c8bbca39cf47afb6512ad3fb8a185cff4f"}
Oct 31 03:51:34.176084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.175659    5831 scope.go:122] "RemoveContainer" containerID="13b48afe81143e2a1d65942e413fe469222000eeb1245809a57ecde980a79177"
Oct 31 03:51:34.176084 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.175596    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"
Oct 31 03:51:34.177720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.177584    5831 generic.go:356] "Generic (PLEG): container finished" podID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" containerID="94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009" exitCode=0
Oct 31 03:51:34.177720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.177643    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" event={"ID":"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f","Type":"ContainerDied","Data":"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"}
Oct 31 03:51:34.177720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.177686    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s" event={"ID":"7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f","Type":"ContainerDied","Data":"e90d6336bb32f971d183ac08cc711791569a054c8cf1d6f10b083efbb54fd2a5"}
Oct 31 03:51:34.177720 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.177694    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"
Oct 31 03:51:34.188767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.188747    5831 scope.go:122] "RemoveContainer" containerID="94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"
Oct 31 03:51:34.200073 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.200049    5831 scope.go:122] "RemoveContainer" containerID="94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"
Oct 31 03:51:34.200449 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:34.200422    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009\": container with ID starting with 94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009 not found: ID does not exist" containerID="94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"
Oct 31 03:51:34.200584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.200461    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009"} err="failed to get container status \"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009\": rpc error: code = NotFound desc = could not find container \"94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009\": container with ID starting with 94b7e28b7bfb61c88b61dde4028bebff465b89e8aba599423149cc6ce3862009 not found: ID does not exist"
Oct 31 03:51:34.201277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.201258    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 03:51:34.201964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.201949    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" containerName="payload-pre-processing"
Oct 31 03:51:34.202034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.201966    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" containerName="payload-pre-processing"
Oct 31 03:51:34.202034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.201997    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f9d0b2ed-2381-49d5-8309-02412b2b3824" containerName="payload-processing"
Oct 31 03:51:34.202034 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.202002    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d0b2ed-2381-49d5-8309-02412b2b3824" containerName="payload-processing"
Oct 31 03:51:34.202145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.202126    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" containerName="payload-pre-processing"
Oct 31 03:51:34.202145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.202140    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d0b2ed-2381-49d5-8309-02412b2b3824" containerName="payload-processing"
Oct 31 03:51:34.207281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.207267    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.255418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.255372    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-93f411-serving-cert\""
Oct 31 03:51:34.257614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.257585    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 03:51:34.258819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.258794    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbs4b\" (UniqueName: \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.258962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.258925    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.259022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.258986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.259086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.259023    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.350502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.350408    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:51:34.357678 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.357645    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14600f-79c884495f-zj52s"]
Oct 31 03:51:34.360612 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.360589    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.360776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.360664    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.360776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.360708    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.360913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.360826    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbs4b\" (UniqueName: \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.360913 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:34.360890    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-serving-cert: secret "maas-api-e2e-shared-a-w4-93f411-serving-cert" not found
Oct 31 03:51:34.361030 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:34.360961    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls podName:b962db6c-ba7e-4f3e-91a7-76187df94545 nodeName:}" failed. No retries permitted until 2026-10-31 03:51:34.86093972 +0000 UTC m=+2644.875501422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls") pod "maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" (UID: "b962db6c-ba7e-4f3e-91a7-76187df94545") : secret "maas-api-e2e-shared-a-w4-93f411-serving-cert" not found
Oct 31 03:51:34.361507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.361486    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.363310 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.363285    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.416794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.416761    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbs4b\" (UniqueName: \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.578509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.578465    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:51:34.583540 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.583496    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14600f-5f64db76dd-hqbfz"]
Oct 31 03:51:34.877611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.877571    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:34.880210 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:34.880185    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:35.122113 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:35.122074    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:35.503798 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:35.503774    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 03:51:35.504503 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:51:35.504480    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb962db6c_ba7e_4f3e_91a7_76187df94545.slice/crio-bd9f102e3267d0c6418c9cc5cce33f160f2e842c4526dd4f11030725675c1611 WatchSource:0}: Error finding container bd9f102e3267d0c6418c9cc5cce33f160f2e842c4526dd4f11030725675c1611: Status 404 returned error can't find the container with id bd9f102e3267d0c6418c9cc5cce33f160f2e842c4526dd4f11030725675c1611
Oct 31 03:51:36.136158 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:36.136124    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f" path="/var/lib/kubelet/pods/7287dd64-1dc1-4fbb-ade0-4d2fa7e4331f/volumes"
Oct 31 03:51:36.136595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:36.136581    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d0b2ed-2381-49d5-8309-02412b2b3824" path="/var/lib/kubelet/pods/f9d0b2ed-2381-49d5-8309-02412b2b3824/volumes"
Oct 31 03:51:36.224311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:36.224277    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" event={"ID":"b962db6c-ba7e-4f3e-91a7-76187df94545","Type":"ContainerStarted","Data":"bd9f102e3267d0c6418c9cc5cce33f160f2e842c4526dd4f11030725675c1611"}
Oct 31 03:51:37.235096 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:37.235062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" event={"ID":"b962db6c-ba7e-4f3e-91a7-76187df94545","Type":"ContainerStarted","Data":"5842ea8de9f15946947b09d236cb61ef131bade4d2465a1f976536a2a1751d89"}
Oct 31 03:51:37.235613 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:37.235201    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:37.363566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:37.363280    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" podStartSLOduration=2.468180305 podStartE2EDuration="3.363254829s" totalImagesPullingTime="895.074524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:37.357227891 +0000 UTC m=+2647.371789601" watchObservedRunningTime="2026-10-31 03:51:37.363254829 +0000 UTC m=+2647.377816534"
Oct 31 03:51:41.081164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:41.081125    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"]
Oct 31 03:51:41.085454 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:41.085425    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc088a5-s4l2d"]
Oct 31 03:51:42.144956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:42.144919    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f05d7a-d8c9-40cd-bc38-0f4db8d95241" path="/var/lib/kubelet/pods/e3f05d7a-d8c9-40cd-bc38-0f4db8d95241/volumes"
Oct 31 03:51:42.274046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:42.274007    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:51:42.274305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:42.274252    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" containerID="cri-o://38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1" gracePeriod=30
Oct 31 03:51:42.979496 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:42.979452    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:51:43.250967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:43.250888    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 03:51:44.825350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:44.825314    5831 patch_prober.go:29] interesting pod/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:51:44.825872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:44.825388    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:51:46.507898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:46.507855    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:51:47.561792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.561754    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:51:47.728744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.728691    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.728977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.728770    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.728970    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:47.729061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729053    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729100    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729388    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729444    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729192    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729576    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729634    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729635    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:47.729772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729676    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\") pod \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\" (UID: \"6ba1feb5-0de9-4d73-a811-ccac25fcce25\") "
Oct 31 03:51:47.730225 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.729927    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:47.730460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.730258    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.730460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.730311    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.730460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.730324    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.730460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.730337    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.730460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.730354    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.732910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.732883    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:51:47.734107 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.734077    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:47.735329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.735308    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "kube-api-access-sp7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:47.735649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.735624    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy" pod "6ba1feb5-0de9-4d73-a811-ccac25fcce25" (UID: "6ba1feb5-0de9-4d73-a811-ccac25fcce25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:47.832007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.831966    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.832007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.832003    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.832249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.832018    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp7h7\" (UniqueName: \"kubernetes.io/projected/6ba1feb5-0de9-4d73-a811-ccac25fcce25-kube-api-access-sp7h7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:47.832249 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:47.832032    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba1feb5-0de9-4d73-a811-ccac25fcce25-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:51:48.377253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.377214    5831 generic.go:356] "Generic (PLEG): container finished" podID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerID="38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1" exitCode=0
Oct 31 03:51:48.377433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.377269    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"
Oct 31 03:51:48.377433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.377287    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerDied","Data":"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"}
Oct 31 03:51:48.377433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.377337    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p" event={"ID":"6ba1feb5-0de9-4d73-a811-ccac25fcce25","Type":"ContainerDied","Data":"e4d0bab86036ec3e9219a92f68d573a14b552dde68855b7c2639d4db5de35c1f"}
Oct 31 03:51:48.377433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.377363    5831 scope.go:122] "RemoveContainer" containerID="38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"
Oct 31 03:51:48.420416 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.420211    5831 scope.go:122] "RemoveContainer" containerID="f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"
Oct 31 03:51:48.431322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.431303    5831 scope.go:122] "RemoveContainer" containerID="38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"
Oct 31 03:51:48.431617 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:48.431592    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1\": container with ID starting with 38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1 not found: ID does not exist" containerID="38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"
Oct 31 03:51:48.431681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.431625    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1"} err="failed to get container status \"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1\": rpc error: code = NotFound desc = could not find container \"38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1\": container with ID starting with 38a4f3d67484323db90227fbe256bc5721f6532bbc0cdfe516d1b331df3f18b1 not found: ID does not exist"
Oct 31 03:51:48.431681 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.431641    5831 scope.go:122] "RemoveContainer" containerID="f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"
Oct 31 03:51:48.431892 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:48.431877    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40\": container with ID starting with f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40 not found: ID does not exist" containerID="f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"
Oct 31 03:51:48.431935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.431898    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40"} err="failed to get container status \"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40\": rpc error: code = NotFound desc = could not find container \"f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40\": container with ID starting with f6b1d317d1155f831fea9a60bbb0e9a609bd26e0e4df7acc408d92f7cbe60b40 not found: ID does not exist"
Oct 31 03:51:48.532808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.532771    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:51:48.582563 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:48.579083    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-14600f-openshift-default-56585bb4b9-jt97p"]
Oct 31 03:51:50.135963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:50.135930    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" path="/var/lib/kubelet/pods/6ba1feb5-0de9-4d73-a811-ccac25fcce25/volumes"
Oct 31 03:51:55.678859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.678818    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 03:51:55.679903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.679880    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:51:55.679975 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.679906    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:51:55.680125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.680111    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:51:55.680167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.680133    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:51:55.686674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.686650    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.736702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.736654    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 03:51:55.761098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.761062    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.761294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.761120    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.761294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.761146    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz68c\" (UniqueName: \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.761294 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.761238    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.774842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.774809    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-861dfd-serving-cert\""
Oct 31 03:51:55.862239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.862203    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.862449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.862257    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.862449 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:55.862346    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-serving-cert: secret "maas-api-e2e-shared-a-w1-861dfd-serving-cert" not found
Oct 31 03:51:55.862449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.862370    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz68c\" (UniqueName: \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.862449 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:51:55.862402    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls podName:d48baeb3-45e6-4f02-8c97-0701d80de3ae nodeName:}" failed. No retries permitted until 2026-10-31 03:51:56.36238797 +0000 UTC m=+2666.376949657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls") pod "maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" (UID: "d48baeb3-45e6-4f02-8c97-0701d80de3ae") : secret "maas-api-e2e-shared-a-w1-861dfd-serving-cert" not found
Oct 31 03:51:55.862449 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.862440    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.863002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.862978    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.865307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.865276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:55.916130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:55.916092    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz68c\" (UniqueName: \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:56.371273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:56.371234    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:56.373932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:56.373904    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:56.604142 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:56.604104    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:57.000132 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:56.997341    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 03:51:57.489108 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.489073    5831 generic.go:356] "Generic (PLEG): container finished" podID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerID="4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3" exitCode=0
Oct 31 03:51:57.489351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.489111    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerDied","Data":"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"}
Oct 31 03:51:57.489603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.489579    5831 scope.go:122] "RemoveContainer" containerID="4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"
Oct 31 03:51:57.491116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.491088    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" event={"ID":"d48baeb3-45e6-4f02-8c97-0701d80de3ae","Type":"ContainerStarted","Data":"cf6b52c2c4066c01037723975d8438bdece8742aad85418876a4a7571f9a399f"}
Oct 31 03:51:57.491252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.491206    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:51:57.491307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.491287    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:51:57.491365 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:57.491349    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:51:58.516109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:58.516071    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" event={"ID":"d48baeb3-45e6-4f02-8c97-0701d80de3ae","Type":"ContainerStarted","Data":"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"}
Oct 31 03:51:58.516673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:58.516152    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:51:58.518820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:58.518798    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerStarted","Data":"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"}
Oct 31 03:51:58.624121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:58.624063    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" podStartSLOduration=2.681484121 podStartE2EDuration="3.624050533s" totalImagesPullingTime="942.566412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:58.622074558 +0000 UTC m=+2668.636636265" watchObservedRunningTime="2026-10-31 03:51:58.624050533 +0000 UTC m=+2668.638612245"
Oct 31 03:51:59.182796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:59.182703    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:51:59.184241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:59.184211    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 31 03:51:59.184474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:59.184443    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 31 03:51:59.850635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:51:59.850599    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:52:00.184558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:00.183886    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 31 03:52:00.184558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:00.183948    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 31 03:52:01.183354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:01.182921    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 31 03:52:01.183354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:01.182980    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 31 03:52:02.183556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:02.181701    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 31 03:52:02.183556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:02.181778    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 31 03:52:03.186934 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:03.186896    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 31 03:52:03.187576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:03.187521    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 31 03:52:04.185897 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:04.185868    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:52:04.187268 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:04.187243    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 03:52:04.532905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:04.532832    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 03:52:04.968581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:04.968548    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:52:05.083238 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:05.079246    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:52:09.969628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:09.969569    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" podUID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" containerName="payload-processing" containerID="cri-o://40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e" gracePeriod=25
Oct 31 03:52:10.082621 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.082584    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" podUID="67c74d1e-340f-4300-86df-ee34b878c46d" containerName="payload-pre-processing" containerID="cri-o://8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c" gracePeriod=25
Oct 31 03:52:10.267407 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.267360    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:52:10.329178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.329134    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\" (UniqueName: \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\") pod \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") "
Oct 31 03:52:10.329356 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.329271    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\") pod \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\" (UID: \"ac691f35-6d32-4809-ab1e-ebc664b0a7d0\") "
Oct 31 03:52:10.330277 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.330240    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume" pod "ac691f35-6d32-4809-ab1e-ebc664b0a7d0" (UID: "ac691f35-6d32-4809-ab1e-ebc664b0a7d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:10.335069 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.335045    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5" pod "ac691f35-6d32-4809-ab1e-ebc664b0a7d0" (UID: "ac691f35-6d32-4809-ab1e-ebc664b0a7d0"). InnerVolumeSpecName "kube-api-access-5qhf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:10.371609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.371581    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:52:10.431288 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431257    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\") pod \"67c74d1e-340f-4300-86df-ee34b878c46d\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") "
Oct 31 03:52:10.431499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431341    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\" (UniqueName: \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\") pod \"67c74d1e-340f-4300-86df-ee34b878c46d\" (UID: \"67c74d1e-340f-4300-86df-ee34b878c46d\") "
Oct 31 03:52:10.431672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431650    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume" pod "67c74d1e-340f-4300-86df-ee34b878c46d" (UID: "67c74d1e-340f-4300-86df-ee34b878c46d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:10.431811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431795    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:10.431859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431813    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c74d1e-340f-4300-86df-ee34b878c46d-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:10.431859 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.431824    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhf5\" (UniqueName: \"kubernetes.io/projected/ac691f35-6d32-4809-ab1e-ebc664b0a7d0-kube-api-access-5qhf5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:10.433688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.433667    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v" pod "67c74d1e-340f-4300-86df-ee34b878c46d" (UID: "67c74d1e-340f-4300-86df-ee34b878c46d"). InnerVolumeSpecName "kube-api-access-6j76v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:10.533183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.533082    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j76v\" (UniqueName: \"kubernetes.io/projected/67c74d1e-340f-4300-86df-ee34b878c46d-kube-api-access-6j76v\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:10.672822 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.672783    5831 generic.go:356] "Generic (PLEG): container finished" podID="67c74d1e-340f-4300-86df-ee34b878c46d" containerID="8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c" exitCode=0
Oct 31 03:52:10.672999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.672831    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"
Oct 31 03:52:10.672999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.672866    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" event={"ID":"67c74d1e-340f-4300-86df-ee34b878c46d","Type":"ContainerDied","Data":"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"}
Oct 31 03:52:10.672999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.672921    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7" event={"ID":"67c74d1e-340f-4300-86df-ee34b878c46d","Type":"ContainerDied","Data":"8c06757ef67f5966a4acd7faf26ae513963d3ef1df225d3e810bf3c712c10114"}
Oct 31 03:52:10.672999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.672945    5831 scope.go:122] "RemoveContainer" containerID="8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"
Oct 31 03:52:10.674658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.674633    5831 generic.go:356] "Generic (PLEG): container finished" podID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" containerID="40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e" exitCode=0
Oct 31 03:52:10.674783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.674689    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" event={"ID":"ac691f35-6d32-4809-ab1e-ebc664b0a7d0","Type":"ContainerDied","Data":"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"}
Oct 31 03:52:10.674783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.674718    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"
Oct 31 03:52:10.674985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.674722    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8" event={"ID":"ac691f35-6d32-4809-ab1e-ebc664b0a7d0","Type":"ContainerDied","Data":"e8e2b63cea7979ffeda69efdc14ff2f04f5ea7c6b9dbb94e7838c6b9a3b60f89"}
Oct 31 03:52:10.686224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.686209    5831 scope.go:122] "RemoveContainer" containerID="8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"
Oct 31 03:52:10.686505 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:10.686488    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c\": container with ID starting with 8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c not found: ID does not exist" containerID="8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"
Oct 31 03:52:10.686620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.686510    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c"} err="failed to get container status \"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c\": rpc error: code = NotFound desc = could not find container \"8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c\": container with ID starting with 8d5c8f230241a684a949345d5e17bda34b40ed0bc09a86daafcde12df6c7153c not found: ID does not exist"
Oct 31 03:52:10.686620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.686543    5831 scope.go:122] "RemoveContainer" containerID="40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"
Oct 31 03:52:10.697675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.697656    5831 scope.go:122] "RemoveContainer" containerID="40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"
Oct 31 03:52:10.697958 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:10.697932    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e\": container with ID starting with 40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e not found: ID does not exist" containerID="40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"
Oct 31 03:52:10.698053 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.697961    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e"} err="failed to get container status \"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e\": rpc error: code = NotFound desc = could not find container \"40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e\": container with ID starting with 40eb056d78407a9b97393bab62f29b41c60f0642ccfdb8d0f24eaa546287610e not found: ID does not exist"
Oct 31 03:52:10.826723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.826639    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:52:10.835675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.835641    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de2877-b45b7ff7-lqwp8"]
Oct 31 03:52:10.938962 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.938931    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:52:10.945491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:10.945464    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de2877-67559ffdb7-thrn7"]
Oct 31 03:52:12.138300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:12.138266    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c74d1e-340f-4300-86df-ee34b878c46d" path="/var/lib/kubelet/pods/67c74d1e-340f-4300-86df-ee34b878c46d/volumes"
Oct 31 03:52:12.138838 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:12.138801    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" path="/var/lib/kubelet/pods/ac691f35-6d32-4809-ab1e-ebc664b0a7d0/volumes"
Oct 31 03:52:12.139178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:12.139161    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:52:12.139370 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:12.139352    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy" containerID="cri-o://689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006" gracePeriod=30
Oct 31 03:52:12.980777 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:12.980719    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:52:14.469788 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.469745    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 03:52:14.470628 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470606    5831 cpu_manager.go:414] "containerMap: removing container" podUID="67c74d1e-340f-4300-86df-ee34b878c46d" containerName="payload-pre-processing"
Oct 31 03:52:14.470747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470629    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c74d1e-340f-4300-86df-ee34b878c46d" containerName="payload-pre-processing"
Oct 31 03:52:14.470747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470676    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:52:14.470747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470686    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba1feb5-0de9-4d73-a811-ccac25fcce25" containerName="istio-proxy"
Oct 31 03:52:14.470747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470730    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" containerName="payload-processing"
Oct 31 03:52:14.470747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470737    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" containerName="payload-processing"
Oct 31 03:52:14.471016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470901    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac691f35-6d32-4809-ab1e-ebc664b0a7d0" containerName="payload-processing"
Oct 31 03:52:14.471016 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.470918    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c74d1e-340f-4300-86df-ee34b878c46d" containerName="payload-pre-processing"
Oct 31 03:52:14.484584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.476248    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.556589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.550253    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ab9b62-openshift-default-dockercfg-85nfh\""
Oct 31 03:52:14.694384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.680474    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.695043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.694944    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.695353 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.695322    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.695658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.695593    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvcx\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.696082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.696061    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.696479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.696458    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.696858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.696825    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.697322 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.697181    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.697499 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.697481    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.702705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.702670    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800330    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800410    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800467    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800510    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800563    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800619    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800685    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800745    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.800773    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvcx\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.802189    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.802477    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.802746    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.804307    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.805070 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.804654    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.852515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.852459    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.853410 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.853378    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.855841 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.855818    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvcx\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:14.862332 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:14.862300    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\") pod \"e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") " pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:15.111790 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.111696    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:15.543090 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:15.543047    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb69b913_b751_4968_b216_4358ea07a83f.slice/crio-aa6f3994155e71bf21387639740bfe90fa6ecd1f6c255bd5758125dd917ea5e1 WatchSource:0}: Error finding container aa6f3994155e71bf21387639740bfe90fa6ecd1f6c255bd5758125dd917ea5e1: Status 404 returned error can't find the container with id aa6f3994155e71bf21387639740bfe90fa6ecd1f6c255bd5758125dd917ea5e1
Oct 31 03:52:15.543569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.543472    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 03:52:15.545846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.545813    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:15.545956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.545877    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:15.545956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.545908    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:15.783662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.783607    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerStarted","Data":"4c6aaea2480fb21d3f224f9046cb49ee5ff00f4a3c3be51e6677e19b1019e5f2"}
Oct 31 03:52:15.783876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.783716    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerStarted","Data":"aa6f3994155e71bf21387639740bfe90fa6ecd1f6c255bd5758125dd917ea5e1"}
Oct 31 03:52:15.893771 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:15.893656    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podStartSLOduration=1.893641369 podStartE2EDuration="1.893641369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:15.891735722 +0000 UTC m=+2685.906297434" watchObservedRunningTime="2026-10-31 03:52:15.893641369 +0000 UTC m=+2685.908203077"
Oct 31 03:52:16.112772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:16.112735    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:16.114755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:16.114726    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:52:16.114913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:16.114778    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:52:17.117842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.117804    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:17.421863 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.421838    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:52:17.540960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.540926    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.540984    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541127    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541171 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541166    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541169    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:17.541351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541281    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541317    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541350    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541389    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541393    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:17.541507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541405    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:17.541507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541483    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\") pod \"9fbd4482-dc64-44e8-869a-be9f06d41363\" (UID: \"9fbd4482-dc64-44e8-869a-be9f06d41363\") "
Oct 31 03:52:17.541839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.541815    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:17.542129 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542097    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:17.542298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542276    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.542371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542304    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.542371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542320    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.542371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542336    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.542371 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.542350    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd4482-dc64-44e8-869a-be9f06d41363-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.543943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.543913    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:17.544160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.544138    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:17.544342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.544316    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:17.544693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.544673    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9" pod "9fbd4482-dc64-44e8-869a-be9f06d41363" (UID: "9fbd4482-dc64-44e8-869a-be9f06d41363"). InnerVolumeSpecName "kube-api-access-h7rv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:17.643387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.643309    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.643387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.643337    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.643387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.643347    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rv9\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-kube-api-access-h7rv9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.643387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.643356    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd4482-dc64-44e8-869a-be9f06d41363-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:17.803433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.803391    5831 generic.go:356] "Generic (PLEG): container finished" podID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerID="689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006" exitCode=0
Oct 31 03:52:17.803665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.803517    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"
Oct 31 03:52:17.803665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.803590    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerDied","Data":"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"}
Oct 31 03:52:17.803791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.803664    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6" event={"ID":"9fbd4482-dc64-44e8-869a-be9f06d41363","Type":"ContainerDied","Data":"61aaa7a3f92034b72e151f703e7d8aeb5bfc53543e7457b18fae194b24d43891"}
Oct 31 03:52:17.803791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.803697    5831 scope.go:122] "RemoveContainer" containerID="689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"
Oct 31 03:52:17.804169 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.804147    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:17.805581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.805501    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:52:17.817840 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.817818    5831 scope.go:122] "RemoveContainer" containerID="7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"
Oct 31 03:52:17.829812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.829791    5831 scope.go:122] "RemoveContainer" containerID="689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"
Oct 31 03:52:17.830167 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:17.830145    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006\": container with ID starting with 689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006 not found: ID does not exist" containerID="689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"
Oct 31 03:52:17.830252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.830178    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006"} err="failed to get container status \"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006\": rpc error: code = NotFound desc = could not find container \"689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006\": container with ID starting with 689a33df2d94df94a25d4e658c56df50c9fa3f242c4baf426d15e663ab21a006 not found: ID does not exist"
Oct 31 03:52:17.830252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.830196    5831 scope.go:122] "RemoveContainer" containerID="7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"
Oct 31 03:52:17.830542 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:17.830505    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0\": container with ID starting with 7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0 not found: ID does not exist" containerID="7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"
Oct 31 03:52:17.830615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:17.830572    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0"} err="failed to get container status \"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0\": rpc error: code = NotFound desc = could not find container \"7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0\": container with ID starting with 7d717f5fc109a0f8930afaa244ae660507255b7ac646c4cca50058f6703f59f0 not found: ID does not exist"
Oct 31 03:52:18.213553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:18.207292    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:52:18.214082 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:18.213695    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca15df-openshift-default-5bc6987489-zhkm6"]
Oct 31 03:52:18.259551 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:18.239175    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:52:20.136970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:20.136935    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" path="/var/lib/kubelet/pods/9fbd4482-dc64-44e8-869a-be9f06d41363/volumes"
Oct 31 03:52:22.656556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.652552    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 03:52:22.656556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.653675    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:22.656556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.653695    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:22.656556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.653872    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:22.675626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.675595    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.695439 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.695402    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 03:52:22.748816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.748741    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ab9b62\""
Oct 31 03:52:22.777870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.777836    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ab9b62-dockercfg-9tjsf\""
Oct 31 03:52:22.843167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.842909    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvtb\" (UniqueName: \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.843167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.843036    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.944734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.944698    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvtb\" (UniqueName: \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.944958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.944785    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.945562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.945522    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:22.998367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:22.998322    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvtb\" (UniqueName: \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\") pod \"payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:23.295618 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.295483    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:23.530564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.530491    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:52:23.530797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.530760    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" podUID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" containerName="maas-api" containerID="cri-o://6f2eb0ba3c105b4e8b344878a0a1b1accdab294a3f3752a976dcba4394711bc3" gracePeriod=30
Oct 31 03:52:23.692969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.692864    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 03:52:23.698204 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:23.698146    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7565c1b3_2899_44b4_981b_bf9d2e8a06d8.slice/crio-36b0127ed2a686157cd6ccbf831521c51ba51343b7cf022134bb375d3ecd1d31 WatchSource:0}: Error finding container 36b0127ed2a686157cd6ccbf831521c51ba51343b7cf022134bb375d3ecd1d31: Status 404 returned error can't find the container with id 36b0127ed2a686157cd6ccbf831521c51ba51343b7cf022134bb375d3ecd1d31
Oct 31 03:52:23.931469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.931437    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" containerID="6f2eb0ba3c105b4e8b344878a0a1b1accdab294a3f3752a976dcba4394711bc3" exitCode=0
Oct 31 03:52:23.931611 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.931595    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" event={"ID":"dc8c4660-090a-4eaf-b8a2-91de9e9032ec","Type":"ContainerDied","Data":"6f2eb0ba3c105b4e8b344878a0a1b1accdab294a3f3752a976dcba4394711bc3"}
Oct 31 03:52:23.936910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.936738    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" event={"ID":"7565c1b3-2899-44b4-981b-bf9d2e8a06d8","Type":"ContainerStarted","Data":"36b0127ed2a686157cd6ccbf831521c51ba51343b7cf022134bb375d3ecd1d31"}
Oct 31 03:52:23.943395 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:23.943369    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:52:24.059163 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.059134    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"]
Oct 31 03:52:24.061024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.060964    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4b1a6-6s5wx"]
Oct 31 03:52:24.101029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.100996    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\") pod \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") "
Oct 31 03:52:24.101879 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.101855    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\" (UniqueName: \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\") pod \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") "
Oct 31 03:52:24.102049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.102033    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\") pod \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") "
Oct 31 03:52:24.102342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.102325    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\") pod \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\" (UID: \"dc8c4660-090a-4eaf-b8a2-91de9e9032ec\") "
Oct 31 03:52:24.104218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.104193    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca" pod "dc8c4660-090a-4eaf-b8a2-91de9e9032ec" (UID: "dc8c4660-090a-4eaf-b8a2-91de9e9032ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:24.139086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.139026    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst" pod "dc8c4660-090a-4eaf-b8a2-91de9e9032ec" (UID: "dc8c4660-090a-4eaf-b8a2-91de9e9032ec"). InnerVolumeSpecName "kube-api-access-lcgst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:24.139380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.139344    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs" pod "dc8c4660-090a-4eaf-b8a2-91de9e9032ec" (UID: "dc8c4660-090a-4eaf-b8a2-91de9e9032ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:24.144664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.144626    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls" pod "dc8c4660-090a-4eaf-b8a2-91de9e9032ec" (UID: "dc8c4660-090a-4eaf-b8a2-91de9e9032ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:24.155556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.154028    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de38b6d-be48-47d1-9a11-d3f1b2c817bc" path="/var/lib/kubelet/pods/5de38b6d-be48-47d1-9a11-d3f1b2c817bc/volumes"
Oct 31 03:52:24.204657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.204626    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.204657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.204659    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.204916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.204675    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcgst\" (UniqueName: \"kubernetes.io/projected/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-kube-api-access-lcgst\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.204916 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.204699    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c4660-090a-4eaf-b8a2-91de9e9032ec-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.947032 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.946999    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"
Oct 31 03:52:24.947504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.947021    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk" event={"ID":"dc8c4660-090a-4eaf-b8a2-91de9e9032ec","Type":"ContainerDied","Data":"bd966abbc80c8078864b0ec9d62fed17d7097746fe5844afb4cc5d03ab75f44a"}
Oct 31 03:52:24.947504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.947130    5831 scope.go:122] "RemoveContainer" containerID="6f2eb0ba3c105b4e8b344878a0a1b1accdab294a3f3752a976dcba4394711bc3"
Oct 31 03:52:24.949390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.949364    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" event={"ID":"7565c1b3-2899-44b4-981b-bf9d2e8a06d8","Type":"ContainerStarted","Data":"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"}
Oct 31 03:52:24.949599 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:24.949581    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:25.103267 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:25.103201    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" podStartSLOduration=3.103181133 podStartE2EDuration="3.103181133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:25.101773228 +0000 UTC m=+2695.116334933" watchObservedRunningTime="2026-10-31 03:52:25.103181133 +0000 UTC m=+2695.117742845"
Oct 31 03:52:25.211014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:25.210868    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:52:25.235550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:25.217790    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-de2877-8644874d59-dn4dk"]
Oct 31 03:52:26.137097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:26.137053    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" path="/var/lib/kubelet/pods/dc8c4660-090a-4eaf-b8a2-91de9e9032ec/volumes"
Oct 31 03:52:29.775662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:29.744117    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:52:29.775662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:29.744388    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy" containerID="cri-o://c563e455284ae145038b05f1e6c5c4009476e89564ef39a12c9f0abfc4bf9db7" gracePeriod=30
Oct 31 03:52:35.048242 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.048213    5831 generic.go:356] "Generic (PLEG): container finished" podID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerID="c563e455284ae145038b05f1e6c5c4009476e89564ef39a12c9f0abfc4bf9db7" exitCode=0
Oct 31 03:52:35.048949 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.048719    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerDied","Data":"c563e455284ae145038b05f1e6c5c4009476e89564ef39a12c9f0abfc4bf9db7"}
Oct 31 03:52:35.049228 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.049210    5831 scope.go:122] "RemoveContainer" containerID="9601d6dd5267e5ccf56596a978437563d1a4f3ad66cf5763d03560dcfadbc5a8"
Oct 31 03:52:35.148979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.148947    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:52:35.282177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282138    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282256    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282302    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282331    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282379    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282424    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282454    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282546    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282609    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\") pod \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\" (UID: \"b00cac98-d2a5-4707-ad96-53bc721e6e4f\") "
Oct 31 03:52:35.282977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.282903    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:35.283054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283034    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:35.283120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283052    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:35.283203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283183    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:35.283270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283232    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:35.283343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283309    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.283343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283332    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.283451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283348    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.283451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.283361    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.285847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.285823    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "kube-api-access-647fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:35.285976 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.285866    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:35.286026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.285994    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:35.286064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.286054    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token" pod "b00cac98-d2a5-4707-ad96-53bc721e6e4f" (UID: "b00cac98-d2a5-4707-ad96-53bc721e6e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:35.389669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.389517    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.389669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.389577    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.389669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.389595    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647fq\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-kube-api-access-647fq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.389669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.389610    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.389669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.389624    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00cac98-d2a5-4707-ad96-53bc721e6e4f-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:35.973080 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:35.973047    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 03:52:36.059912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.059881    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"
Oct 31 03:52:36.059912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.059897    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc" event={"ID":"b00cac98-d2a5-4707-ad96-53bc721e6e4f","Type":"ContainerDied","Data":"714c79b5da0bfe5965ff61e0aec068344c4f78b06d24f61f70c037dc535a4b5b"}
Oct 31 03:52:36.060360 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.059963    5831 scope.go:122] "RemoveContainer" containerID="c563e455284ae145038b05f1e6c5c4009476e89564ef39a12c9f0abfc4bf9db7"
Oct 31 03:52:36.186595 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.186560    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:52:36.232421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.232342    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-de2877-openshift-default-b9fbff7d9-ngwrc"]
Oct 31 03:52:36.662747 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.662661    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 03:52:36.663348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663333    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:36.663405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663351    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:36.663405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663360    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663366    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663388    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" containerName="maas-api"
Oct 31 03:52:36.663405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663394    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" containerName="maas-api"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663411    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663417    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663553    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663607    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8c4660-090a-4eaf-b8a2-91de9e9032ec" containerName="maas-api"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663626    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" containerName="istio-proxy"
Oct 31 03:52:36.663710 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.663648    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbd4482-dc64-44e8-869a-be9f06d41363" containerName="istio-proxy"
Oct 31 03:52:36.669311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.669291    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.713743 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.713702    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.713960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.713813    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.713960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.713868    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.713995    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plq2\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.714053    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.714135    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.714204    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714305 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.714283    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.714473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.714434    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.724474 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.723593    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 03:52:36.730029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.730001    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-502ec2-openshift-default-dockercfg-9sfll\""
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.821800    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.822669    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.822736    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.823976    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.824603    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plq2\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.825183    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.825320    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.825777    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.831373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.826219    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.826322    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.835736    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.836208    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.836422    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837561 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.837555    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.837992 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.837793    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.852594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.845119    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.907554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.898424    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.916350 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.916270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plq2\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\") pod \"e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") " pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:36.986201 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:36.986169    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:37.416768 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:37.416740    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 03:52:37.417789 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:37.417753    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd389e7f6_79c7_4631_a3d0_51f7989592dd.slice/crio-1963dc59e6f0d79ce35002cc1dd49a85ba2bb786b0d865b9d05b92570ea2a4d0 WatchSource:0}: Error finding container 1963dc59e6f0d79ce35002cc1dd49a85ba2bb786b0d865b9d05b92570ea2a4d0: Status 404 returned error can't find the container with id 1963dc59e6f0d79ce35002cc1dd49a85ba2bb786b0d865b9d05b92570ea2a4d0
Oct 31 03:52:37.420703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:37.420666    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:37.420797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:37.420754    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:37.420839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:37.420795    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:37.908445 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:37.908409    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:52:38.008412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.008375    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:52:38.100503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.100458    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerStarted","Data":"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"}
Oct 31 03:52:38.100690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.100521    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerStarted","Data":"1963dc59e6f0d79ce35002cc1dd49a85ba2bb786b0d865b9d05b92570ea2a4d0"}
Oct 31 03:52:38.139024 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.138992    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00cac98-d2a5-4707-ad96-53bc721e6e4f" path="/var/lib/kubelet/pods/b00cac98-d2a5-4707-ad96-53bc721e6e4f/volumes"
Oct 31 03:52:38.243919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.243857    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podStartSLOduration=2.243837441 podStartE2EDuration="2.243837441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:38.240974079 +0000 UTC m=+2708.255535786" watchObservedRunningTime="2026-10-31 03:52:38.243837441 +0000 UTC m=+2708.258399150"
Oct 31 03:52:38.306185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.306157    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:52:38.308373 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.308355    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:52:38.316001 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.315986    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:52:38.318364 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.318340    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:52:38.319738 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.319720    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:52:38.322791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.322771    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:52:38.986735 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.986691    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:38.992724 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:38.992697    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:39.108485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:39.108455    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:39.109361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:39.109340    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:52:40.930338 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:40.930301    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:52:40.930891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:40.930574    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" podUID="b7118eab-d6d0-4ab6-8266-18ecffa94848" containerName="maas-api" containerID="cri-o://1658fad784364fcb1f4d2f0185466198da342a7f293c8c1b8870f2e6870f9879" gracePeriod=30
Oct 31 03:52:41.131065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.131030    5831 generic.go:356] "Generic (PLEG): container finished" podID="b7118eab-d6d0-4ab6-8266-18ecffa94848" containerID="1658fad784364fcb1f4d2f0185466198da342a7f293c8c1b8870f2e6870f9879" exitCode=0
Oct 31 03:52:41.131065 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.131062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" event={"ID":"b7118eab-d6d0-4ab6-8266-18ecffa94848","Type":"ContainerDied","Data":"1658fad784364fcb1f4d2f0185466198da342a7f293c8c1b8870f2e6870f9879"}
Oct 31 03:52:41.296446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.296417    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:52:41.433015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.432963    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\") pod \"b7118eab-d6d0-4ab6-8266-18ecffa94848\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") "
Oct 31 03:52:41.437779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.435463    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca" pod "b7118eab-d6d0-4ab6-8266-18ecffa94848" (UID: "b7118eab-d6d0-4ab6-8266-18ecffa94848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:41.438264 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.438212    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") pod \"b7118eab-d6d0-4ab6-8266-18ecffa94848\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") "
Oct 31 03:52:41.438873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.438853    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\") pod \"b7118eab-d6d0-4ab6-8266-18ecffa94848\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") "
Oct 31 03:52:41.439471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.439452    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\" (UniqueName: \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\") pod \"b7118eab-d6d0-4ab6-8266-18ecffa94848\" (UID: \"b7118eab-d6d0-4ab6-8266-18ecffa94848\") "
Oct 31 03:52:41.442708 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.442670    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7118eab-d6d0-4ab6-8266-18ecffa94848-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:41.456092 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.456035    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq" pod "b7118eab-d6d0-4ab6-8266-18ecffa94848" (UID: "b7118eab-d6d0-4ab6-8266-18ecffa94848"). InnerVolumeSpecName "kube-api-access-gchxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:41.464548 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.464486    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls" pod "b7118eab-d6d0-4ab6-8266-18ecffa94848" (UID: "b7118eab-d6d0-4ab6-8266-18ecffa94848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:41.470683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.470608    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs" pod "b7118eab-d6d0-4ab6-8266-18ecffa94848" (UID: "b7118eab-d6d0-4ab6-8266-18ecffa94848"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:41.559411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.559243    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchxq\" (UniqueName: \"kubernetes.io/projected/b7118eab-d6d0-4ab6-8266-18ecffa94848-kube-api-access-gchxq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:41.559411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.559276    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:41.559411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:41.559290    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7118eab-d6d0-4ab6-8266-18ecffa94848-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:42.193241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.193207    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"
Oct 31 03:52:42.193782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.193246    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b" event={"ID":"b7118eab-d6d0-4ab6-8266-18ecffa94848","Type":"ContainerDied","Data":"3aaed51108ff1cf5a4f93e0019b413b9e234f6d5ee5e05f73dfb3b85709332a4"}
Oct 31 03:52:42.193782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.193318    5831 scope.go:122] "RemoveContainer" containerID="1658fad784364fcb1f4d2f0185466198da342a7f293c8c1b8870f2e6870f9879"
Oct 31 03:52:42.347894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.347857    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:52:42.394513 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.394478    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2f0bd-5d69b78756-kww7b"]
Oct 31 03:52:42.909685 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.909636    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" podUID="d353fc33-79fb-42e1-a116-34e6fb182555" containerName="payload-processing" containerID="cri-o://45ede22ad12e129c452b4e02e6cc37360c2840cecfef43d2343da9a60843d0d7" gracePeriod=25
Oct 31 03:52:42.982455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.982417    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 03:52:42.983192 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.983171    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b7118eab-d6d0-4ab6-8266-18ecffa94848" containerName="maas-api"
Oct 31 03:52:42.983315 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.983194    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7118eab-d6d0-4ab6-8266-18ecffa94848" containerName="maas-api"
Oct 31 03:52:42.983401 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.983388    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7118eab-d6d0-4ab6-8266-18ecffa94848" containerName="maas-api"
Oct 31 03:52:42.983723 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:52:42.983692    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:52:42.987893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.987870    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989097    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcl6m\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989150    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989305    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989384 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989365    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989485    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989518    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989617    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989675 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989651    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:42.989714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:42.989676    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.010945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.010262    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" podUID="dc92ffe9-dcd1-400f-96ed-13db87829653" containerName="payload-pre-processing" containerID="cri-o://cee79b5498e1908599d0914a4e50e08f47c829060e13a7e1581ed48cd2323cc2" gracePeriod=25
Oct 31 03:52:43.056994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.056957    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c0d50c-openshift-default-dockercfg-zmz4v\""
Oct 31 03:52:43.093889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.093391    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094133    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094208    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094228    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094257    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094283 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094278    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094300    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094329    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcl6m\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094372    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094437    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.094726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.094706    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.095186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.095017    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.100007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.097991    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.100007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.098232    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.100007 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.098768    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.113382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.112896    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 03:52:43.113382 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.113276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.168295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.167707    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.168937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.168909    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcl6m\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\") pod \"e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") " pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.249466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.249212    5831 generic.go:356] "Generic (PLEG): container finished" podID="d353fc33-79fb-42e1-a116-34e6fb182555" containerID="45ede22ad12e129c452b4e02e6cc37360c2840cecfef43d2343da9a60843d0d7" exitCode=0
Oct 31 03:52:43.249466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.249384    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" event={"ID":"d353fc33-79fb-42e1-a116-34e6fb182555","Type":"ContainerDied","Data":"45ede22ad12e129c452b4e02e6cc37360c2840cecfef43d2343da9a60843d0d7"}
Oct 31 03:52:43.253435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.252416    5831 generic.go:356] "Generic (PLEG): container finished" podID="dc92ffe9-dcd1-400f-96ed-13db87829653" containerID="cee79b5498e1908599d0914a4e50e08f47c829060e13a7e1581ed48cd2323cc2" exitCode=0
Oct 31 03:52:43.253435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.252554    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" event={"ID":"dc92ffe9-dcd1-400f-96ed-13db87829653","Type":"ContainerDied","Data":"cee79b5498e1908599d0914a4e50e08f47c829060e13a7e1581ed48cd2323cc2"}
Oct 31 03:52:43.323564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.314338    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:43.366562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.363730    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:52:43.459241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.459219    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:52:43.492938 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.492880    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\") pod \"d353fc33-79fb-42e1-a116-34e6fb182555\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") "
Oct 31 03:52:43.493124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.492949    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\" (UniqueName: \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\") pod \"d353fc33-79fb-42e1-a116-34e6fb182555\" (UID: \"d353fc33-79fb-42e1-a116-34e6fb182555\") "
Oct 31 03:52:43.494994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.494948    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume" pod "d353fc33-79fb-42e1-a116-34e6fb182555" (UID: "d353fc33-79fb-42e1-a116-34e6fb182555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:43.501468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.501431    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw" pod "d353fc33-79fb-42e1-a116-34e6fb182555" (UID: "d353fc33-79fb-42e1-a116-34e6fb182555"). InnerVolumeSpecName "kube-api-access-s8txw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:43.595090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.595051    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\" (UniqueName: \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\") pod \"dc92ffe9-dcd1-400f-96ed-13db87829653\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") "
Oct 31 03:52:43.595289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.595120    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\") pod \"dc92ffe9-dcd1-400f-96ed-13db87829653\" (UID: \"dc92ffe9-dcd1-400f-96ed-13db87829653\") "
Oct 31 03:52:43.595787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.595759    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fc33-79fb-42e1-a116-34e6fb182555-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:43.596075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.595790    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8txw\" (UniqueName: \"kubernetes.io/projected/d353fc33-79fb-42e1-a116-34e6fb182555-kube-api-access-s8txw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:43.596248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.596220    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume" pod "dc92ffe9-dcd1-400f-96ed-13db87829653" (UID: "dc92ffe9-dcd1-400f-96ed-13db87829653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:43.597711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.597685    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k" pod "dc92ffe9-dcd1-400f-96ed-13db87829653" (UID: "dc92ffe9-dcd1-400f-96ed-13db87829653"). InnerVolumeSpecName "kube-api-access-mnr7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:43.696820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.696783    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnr7k\" (UniqueName: \"kubernetes.io/projected/dc92ffe9-dcd1-400f-96ed-13db87829653-kube-api-access-mnr7k\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:43.696820 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.696815    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc92ffe9-dcd1-400f-96ed-13db87829653-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:43.831203 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.831171    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 03:52:43.832217 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:43.832191    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7280b668_d83a_4977_9cdc_1a394ac5b9f7.slice/crio-0179578451fcefe4b282e13ba7e61c6ef51bda36af990ec2c8d0aa61b82905a3 WatchSource:0}: Error finding container 0179578451fcefe4b282e13ba7e61c6ef51bda36af990ec2c8d0aa61b82905a3: Status 404 returned error can't find the container with id 0179578451fcefe4b282e13ba7e61c6ef51bda36af990ec2c8d0aa61b82905a3
Oct 31 03:52:43.834779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.834749    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:43.834874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.834812    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:43.834874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:43.834842    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:52:44.138256 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.138221    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7118eab-d6d0-4ab6-8266-18ecffa94848" path="/var/lib/kubelet/pods/b7118eab-d6d0-4ab6-8266-18ecffa94848/volumes"
Oct 31 03:52:44.268094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.268056    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerStarted","Data":"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"}
Oct 31 03:52:44.268639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.268127    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerStarted","Data":"0179578451fcefe4b282e13ba7e61c6ef51bda36af990ec2c8d0aa61b82905a3"}
Oct 31 03:52:44.269658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.269640    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"
Oct 31 03:52:44.269779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.269686    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt" event={"ID":"d353fc33-79fb-42e1-a116-34e6fb182555","Type":"ContainerDied","Data":"0d0375b61d6cdc60d78e153769cd7830566bf9fcfc5702a1694894e201599297"}
Oct 31 03:52:44.269779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.269742    5831 scope.go:122] "RemoveContainer" containerID="45ede22ad12e129c452b4e02e6cc37360c2840cecfef43d2343da9a60843d0d7"
Oct 31 03:52:44.271944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.271766    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz" event={"ID":"dc92ffe9-dcd1-400f-96ed-13db87829653","Type":"ContainerDied","Data":"e8a2c8b3db97d5f5bfb9c4a81d9e9542fe854d68a6620a9273e9a5bc214f46dc"}
Oct 31 03:52:44.271944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.271784    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"
Oct 31 03:52:44.286807 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.282693    5831 scope.go:122] "RemoveContainer" containerID="cee79b5498e1908599d0914a4e50e08f47c829060e13a7e1581ed48cd2323cc2"
Oct 31 03:52:44.341797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.341768    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:44.347624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.347594    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:44.380480 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.380408    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podStartSLOduration=2.380388075 podStartE2EDuration="2.380388075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:44.376514173 +0000 UTC m=+2714.391075882" watchObservedRunningTime="2026-10-31 03:52:44.380388075 +0000 UTC m=+2714.394949787"
Oct 31 03:52:44.484484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.484440    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:52:44.491467 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.491437    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2f0bd-9f97d6cff-kzhcz"]
Oct 31 03:52:44.600086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.599992    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:52:44.606905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:44.606871    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2f0bd-7b98bf5d5f-jqjpt"]
Oct 31 03:52:45.150552 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.149953    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 03:52:45.150969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.150931    5831 cpu_manager.go:414] "containerMap: removing container" podUID="dc92ffe9-dcd1-400f-96ed-13db87829653" containerName="payload-pre-processing"
Oct 31 03:52:45.150969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.150958    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc92ffe9-dcd1-400f-96ed-13db87829653" containerName="payload-pre-processing"
Oct 31 03:52:45.151135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.151035    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d353fc33-79fb-42e1-a116-34e6fb182555" containerName="payload-processing"
Oct 31 03:52:45.151135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.151043    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d353fc33-79fb-42e1-a116-34e6fb182555" containerName="payload-processing"
Oct 31 03:52:45.151247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.151189    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d353fc33-79fb-42e1-a116-34e6fb182555" containerName="payload-processing"
Oct 31 03:52:45.151247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.151202    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc92ffe9-dcd1-400f-96ed-13db87829653" containerName="payload-pre-processing"
Oct 31 03:52:45.162509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.162473    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.197741 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.197711    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 03:52:45.211405 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.211362    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-502ec2\""
Oct 31 03:52:45.211592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.211512    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-502ec2-dockercfg-zqtq4\""
Oct 31 03:52:45.257564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.250665    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.257564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.250826    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56sv\" (UniqueName: \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.381177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.369800    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.381177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.369930    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56sv\" (UniqueName: \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.381177 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.371182    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.426600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.395584    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 03:52:45.426600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.407284    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.426600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.422613    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 03:52:45.430587 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.430561    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56sv\" (UniqueName: \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\") pod \"payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.449658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.449633    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:45.451180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.451125    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:52:45.471456 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.471415    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.471627 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.471472    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2f7c\" (UniqueName: \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.483776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.483751    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:45.574475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.572918    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.574475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.572966    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2f7c\" (UniqueName: \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.574475 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.574297    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.635633 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.633271    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2f7c\" (UniqueName: \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\") pod \"payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.760579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.760507    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:45.873350 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:45.873312    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57de5143_5837_4403_b4f0_63d7fbd4482b.slice/crio-dc8f7bb7ba3a899f3df9372cb17271bf7c5887db9d46f2f582e9bf13926dd23e WatchSource:0}: Error finding container dc8f7bb7ba3a899f3df9372cb17271bf7c5887db9d46f2f582e9bf13926dd23e: Status 404 returned error can't find the container with id dc8f7bb7ba3a899f3df9372cb17271bf7c5887db9d46f2f582e9bf13926dd23e
Oct 31 03:52:45.873506 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:45.873386    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 03:52:46.045872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.045790    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"]
Oct 31 03:52:46.049876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.049851    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9af1989-bxk9d"]
Oct 31 03:52:46.136328 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.136289    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5800a46c-8d36-4f52-91d1-ca820856e968" path="/var/lib/kubelet/pods/5800a46c-8d36-4f52-91d1-ca820856e968/volumes"
Oct 31 03:52:46.136910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.136886    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d353fc33-79fb-42e1-a116-34e6fb182555" path="/var/lib/kubelet/pods/d353fc33-79fb-42e1-a116-34e6fb182555/volumes"
Oct 31 03:52:46.137352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.137333    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc92ffe9-dcd1-400f-96ed-13db87829653" path="/var/lib/kubelet/pods/dc92ffe9-dcd1-400f-96ed-13db87829653/volumes"
Oct 31 03:52:46.141603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.141577    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 03:52:46.141708 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:46.141687    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22186f4_aee3_49d9_b2b7_753e7d4be148.slice/crio-6b28a1bccb4fab4ef04e2fe1911f4265b256f02cf753a4d27abaac416745fb3d WatchSource:0}: Error finding container 6b28a1bccb4fab4ef04e2fe1911f4265b256f02cf753a4d27abaac416745fb3d: Status 404 returned error can't find the container with id 6b28a1bccb4fab4ef04e2fe1911f4265b256f02cf753a4d27abaac416745fb3d
Oct 31 03:52:46.458104 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.458072    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" event={"ID":"57de5143-5837-4403-b4f0-63d7fbd4482b","Type":"ContainerStarted","Data":"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"}
Oct 31 03:52:46.458622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.458124    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" event={"ID":"57de5143-5837-4403-b4f0-63d7fbd4482b","Type":"ContainerStarted","Data":"dc8f7bb7ba3a899f3df9372cb17271bf7c5887db9d46f2f582e9bf13926dd23e"}
Oct 31 03:52:46.458622 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.458166    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:46.460002 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.459977    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" event={"ID":"b22186f4-aee3-49d9-b2b7-753e7d4be148","Type":"ContainerStarted","Data":"ba2ba04d0db412d5745b9fd4437ba48a6b95d0116cb6dd1b28f6138d0acdf4e6"}
Oct 31 03:52:46.460141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.460014    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" event={"ID":"b22186f4-aee3-49d9-b2b7-753e7d4be148","Type":"ContainerStarted","Data":"6b28a1bccb4fab4ef04e2fe1911f4265b256f02cf753a4d27abaac416745fb3d"}
Oct 31 03:52:46.460218 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.460164    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:46.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.567353    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" podStartSLOduration=1.567339816 podStartE2EDuration="1.567339816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:46.563179554 +0000 UTC m=+2716.577741263" watchObservedRunningTime="2026-10-31 03:52:46.567339816 +0000 UTC m=+2716.581901515"
Oct 31 03:52:46.671300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:46.671238    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" podStartSLOduration=1.671221744 podStartE2EDuration="1.671221744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:46.66771225 +0000 UTC m=+2716.682273941" watchObservedRunningTime="2026-10-31 03:52:46.671221744 +0000 UTC m=+2716.685783452"
Oct 31 03:52:49.565120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:49.565087    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:52:51.082172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.082125    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 03:52:51.095431 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.095393    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.174564 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.174484    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 03:52:51.188417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.187891    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c0d50c-dockercfg-wjtfh\""
Oct 31 03:52:51.188417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.187996    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c0d50c\""
Oct 31 03:52:51.198597 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.198563    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6lff\" (UniqueName: \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.199017 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.198996    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.239176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.239131    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 03:52:51.243991 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.243963    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.286881 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.286849    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 03:52:51.323889 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.323840    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6lff\" (UniqueName: \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.324095 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.324039    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.325042 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.324997    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.377176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.377101    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6lff\" (UniqueName: \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\") pod \"payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.424873 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.424843    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.425061 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.425034    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zf48\" (UniqueName: \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.455224 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.455196    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:51.536205 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.536174    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.536392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.536302    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zf48\" (UniqueName: \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.537154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.537119    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.590079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.590046    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zf48\" (UniqueName: \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\") pod \"payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:51.851363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.850020    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 03:52:51.855333 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:52:51.855300    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f908d42_2b5c_43ec_8c4a_8bb2301424f9.slice/crio-320813e119dd98a6ba9e569f7052bf64079e2b6ff7805edca786c5fd4a3925fd WatchSource:0}: Error finding container 320813e119dd98a6ba9e569f7052bf64079e2b6ff7805edca786c5fd4a3925fd: Status 404 returned error can't find the container with id 320813e119dd98a6ba9e569f7052bf64079e2b6ff7805edca786c5fd4a3925fd
Oct 31 03:52:51.858701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:51.858504    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:52.282091 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.282050    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 03:52:52.513273 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.513245    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" event={"ID":"3e6a16f7-496b-40aa-ac25-1d9588d2edb7","Type":"ContainerStarted","Data":"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"}
Oct 31 03:52:52.513411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.513298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" event={"ID":"3e6a16f7-496b-40aa-ac25-1d9588d2edb7","Type":"ContainerStarted","Data":"db3f830c9e9485edcddf7563a3ecfc12daa3582fc526fb44b658fd9d935d83da"}
Oct 31 03:52:52.513411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.513328    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:52:52.515363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.515336    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" event={"ID":"0f908d42-2b5c-43ec-8c4a-8bb2301424f9","Type":"ContainerStarted","Data":"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"}
Oct 31 03:52:52.515458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.515387    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" event={"ID":"0f908d42-2b5c-43ec-8c4a-8bb2301424f9","Type":"ContainerStarted","Data":"320813e119dd98a6ba9e569f7052bf64079e2b6ff7805edca786c5fd4a3925fd"}
Oct 31 03:52:52.515458 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.515420    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:52:52.620251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.620134    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" podStartSLOduration=1.620120994 podStartE2EDuration="1.620120994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:52.619162027 +0000 UTC m=+2722.633723736" watchObservedRunningTime="2026-10-31 03:52:52.620120994 +0000 UTC m=+2722.634682702"
Oct 31 03:52:52.726048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:52.725991    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" podStartSLOduration=1.725975995 podStartE2EDuration="1.725975995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:52.723893464 +0000 UTC m=+2722.738455171" watchObservedRunningTime="2026-10-31 03:52:52.725975995 +0000 UTC m=+2722.740537703"
Oct 31 03:52:55.044363 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:55.044329    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"]
Oct 31 03:52:55.090397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:55.090362    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58d3705-wr4wd"]
Oct 31 03:52:56.136404 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:56.136371    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9997e0-e6c5-4e59-8fb4-60b576ce9a17" path="/var/lib/kubelet/pods/be9997e0-e6c5-4e59-8fb4-60b576ce9a17/volumes"
Oct 31 03:52:57.471214 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:57.471185    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 03:52:57.471714 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:57.471325    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 03:52:59.901554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.899802    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 03:52:59.925550 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.924392    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955126    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955190    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955238    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955286    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955356    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955412    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955453    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955483    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbz4\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:52:59.955514    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.021592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.012800    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 03:53:00.022385 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.021723    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a5f70c-openshift-default-dockercfg-4rfzl\""
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083138    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083242    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083285    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083314    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbz4\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083343    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083449    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083510    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083581    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.084626 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.083650    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.116636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.104218    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.116636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.112140    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.116636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.112834    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.116636 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.113124    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.118660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.117342    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.118660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.117644    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.138620 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.134178    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.171960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.171881    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbz4\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.171960 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.171911    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\") pod \"e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") " pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.283558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.269273    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:00.748470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.748434    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 03:53:00.784981 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:53:00.784939    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1e9ecf_c40e_4091_9e24_4617a766b04a.slice/crio-44cb9e810d39aba4f8171bde45ce57c250ac2926b75dae07478a134e5c4ecf01 WatchSource:0}: Error finding container 44cb9e810d39aba4f8171bde45ce57c250ac2926b75dae07478a134e5c4ecf01: Status 404 returned error can't find the container with id 44cb9e810d39aba4f8171bde45ce57c250ac2926b75dae07478a134e5c4ecf01
Oct 31 03:53:00.787680 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.787643    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:00.787812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.787733    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:00.787812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:00.787781    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:01.604188 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:01.604149    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerStarted","Data":"2af151d3490945def1a9380bfd00fc46c874e58c4c1dea2e9775799500dfb6d4"}
Oct 31 03:53:01.604744 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:01.604212    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerStarted","Data":"44cb9e810d39aba4f8171bde45ce57c250ac2926b75dae07478a134e5c4ecf01"}
Oct 31 03:53:01.759639 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:01.759583    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podStartSLOduration=2.759567217 podStartE2EDuration="2.759567217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:01.758399274 +0000 UTC m=+2731.772960981" watchObservedRunningTime="2026-10-31 03:53:01.759567217 +0000 UTC m=+2731.774128916"
Oct 31 03:53:02.288776 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:02.288738    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:02.294671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:02.294648    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:02.612852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:02.612770    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:02.613816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:02.613798    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:53:03.046932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:03.046885    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"]
Oct 31 03:53:03.051359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:03.051327    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9738d-kdhj8"]
Oct 31 03:53:03.532477 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:03.532442    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 03:53:03.532979 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:03.532960    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 03:53:04.136193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:04.136161    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c80aded-7843-40c3-b4cc-016fdb0b27fb" path="/var/lib/kubelet/pods/4c80aded-7843-40c3-b4cc-016fdb0b27fb/volumes"
Oct 31 03:53:05.553180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.553147    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"]
Oct 31 03:53:05.559094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.559072    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.609898 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.609869    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"]
Oct 31 03:53:05.650440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.650395    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:53:05.650692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.650488    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:53:05.669829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.669801    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.670018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.669847    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.670137 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.670050    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8lp\" (UniqueName: \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.771312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.771274    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.771524 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.771321    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.771524 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.771412    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8lp\" (UniqueName: \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.772079 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.772055    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.774342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.774317    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.824880 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.824796    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8lp\" (UniqueName: \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\") pod \"maas-api-revoke-sub-e3944492-dpn7b\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:05.875803 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:05.875771    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:06.269721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:06.269693    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"]
Oct 31 03:53:06.272018 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:53:06.271990    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d283016_5cd7_4602_88fb_31f156e52a29.slice/crio-4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00 WatchSource:0}: Error finding container 4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00: Status 404 returned error can't find the container with id 4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00
Oct 31 03:53:06.653665 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:06.653629    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b" event={"ID":"0d283016-5cd7-4602-88fb-31f156e52a29","Type":"ContainerStarted","Data":"4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00"}
Oct 31 03:53:07.665147 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:07.665042    5831 generic.go:356] "Generic (PLEG): container finished" podID="0d283016-5cd7-4602-88fb-31f156e52a29" containerID="c5e971b8ed5889e0e52375974280063ee60e22beb75a8d62ec03a147127a7c1a" exitCode=0
Oct 31 03:53:07.665588 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:07.665260    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b" event={"ID":"0d283016-5cd7-4602-88fb-31f156e52a29","Type":"ContainerDied","Data":"c5e971b8ed5889e0e52375974280063ee60e22beb75a8d62ec03a147127a7c1a"}
Oct 31 03:53:08.896342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.896318    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:08.988131 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.988046    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\") pod \"0d283016-5cd7-4602-88fb-31f156e52a29\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") "
Oct 31 03:53:08.988300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.988263    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\") pod \"0d283016-5cd7-4602-88fb-31f156e52a29\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") "
Oct 31 03:53:08.988359 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.988321    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\" (UniqueName: \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\") pod \"0d283016-5cd7-4602-88fb-31f156e52a29\" (UID: \"0d283016-5cd7-4602-88fb-31f156e52a29\") "
Oct 31 03:53:08.988705 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.988673    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca" pod "0d283016-5cd7-4602-88fb-31f156e52a29" (UID: "0d283016-5cd7-4602-88fb-31f156e52a29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:08.989139 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.989073    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:08.990502 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.990473    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca" pod "0d283016-5cd7-4602-88fb-31f156e52a29" (UID: "0d283016-5cd7-4602-88fb-31f156e52a29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:53:08.990688 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:08.990663    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp" pod "0d283016-5cd7-4602-88fb-31f156e52a29" (UID: "0d283016-5cd7-4602-88fb-31f156e52a29"). InnerVolumeSpecName "kube-api-access-vv8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:09.090414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:09.090376    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8lp\" (UniqueName: \"kubernetes.io/projected/0d283016-5cd7-4602-88fb-31f156e52a29-kube-api-access-vv8lp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:09.090414 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:09.090413    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d283016-5cd7-4602-88fb-31f156e52a29-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:09.712421 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:09.712391    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"
Oct 31 03:53:09.712686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:09.712475    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b" event={"ID":"0d283016-5cd7-4602-88fb-31f156e52a29","Type":"ContainerDied","Data":"4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00"}
Oct 31 03:53:09.712686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:09.712565    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a287c473584c8e3f0c9fb852a8f3aa8bd5825464de0e949449746b09879ef00"
Oct 31 03:53:12.979384 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:53:12.979311    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:53:13.047064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:13.047024    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"]
Oct 31 03:53:13.095516 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:13.095477    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824c927c-4fm6q"]
Oct 31 03:53:14.137097 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:14.137055    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8" path="/var/lib/kubelet/pods/f3cf2aa4-497b-4e1b-a9e2-907574e9b2d8/volumes"
Oct 31 03:53:16.396021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.394057    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"]
Oct 31 03:53:16.396021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.395237    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0d283016-5cd7-4602-88fb-31f156e52a29" containerName="revoke-keys"
Oct 31 03:53:16.396021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.395261    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d283016-5cd7-4602-88fb-31f156e52a29" containerName="revoke-keys"
Oct 31 03:53:16.396021 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.395508    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d283016-5cd7-4602-88fb-31f156e52a29" containerName="revoke-keys"
Oct 31 03:53:16.403973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.403946    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.438011 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.437976    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"]
Oct 31 03:53:16.464901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.464869    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:53:16.465336 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.465288    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:53:16.538825 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.538784    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpfr\" (UniqueName: \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.539241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.539208    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.539361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.539266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.641116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.641078    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpfr\" (UniqueName: \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.641386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.641235    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.641386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.641277    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.642391 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.642362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.644295 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.644266    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.694903 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.694865    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpfr\" (UniqueName: \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\") pod \"maas-api-revoke-sub-19c2b868-88nb6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:16.740375 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:16.740323    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:17.134767 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:53:17.134732    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953bf90c_14d4_4c33_8d48_dfc8bf3351c6.slice/crio-d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30 WatchSource:0}: Error finding container d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30: Status 404 returned error can't find the container with id d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30
Oct 31 03:53:17.136426 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:17.136394    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"]
Oct 31 03:53:17.797208 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:17.797166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6" event={"ID":"953bf90c-14d4-4c33-8d48-dfc8bf3351c6","Type":"ContainerStarted","Data":"508b2f3b2b343686ca21c115102702838aa866837d8ae0f5785666fd90c96232"}
Oct 31 03:53:17.797783 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:17.797238    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6" event={"ID":"953bf90c-14d4-4c33-8d48-dfc8bf3351c6","Type":"ContainerStarted","Data":"d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30"}
Oct 31 03:53:17.905643 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:17.905593    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6" podStartSLOduration=1.905578915 podStartE2EDuration="1.905578915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:17.902348507 +0000 UTC m=+2747.916910230" watchObservedRunningTime="2026-10-31 03:53:17.905578915 +0000 UTC m=+2747.920140622"
Oct 31 03:53:18.051789 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:18.051698    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"]
Oct 31 03:53:18.058051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:18.058021    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e47886-p5qnf"]
Oct 31 03:53:18.141110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:18.141076    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546530ba-d570-4ff5-91b2-3c37a6c76067" path="/var/lib/kubelet/pods/546530ba-d570-4ff5-91b2-3c37a6c76067/volumes"
Oct 31 03:53:18.809261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:18.809225    5831 generic.go:356] "Generic (PLEG): container finished" podID="953bf90c-14d4-4c33-8d48-dfc8bf3351c6" containerID="508b2f3b2b343686ca21c115102702838aa866837d8ae0f5785666fd90c96232" exitCode=0
Oct 31 03:53:18.809842 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:18.809313    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6" event={"ID":"953bf90c-14d4-4c33-8d48-dfc8bf3351c6","Type":"ContainerDied","Data":"508b2f3b2b343686ca21c115102702838aa866837d8ae0f5785666fd90c96232"}
Oct 31 03:53:19.977387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:19.977363    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:20.085811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.085688    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\" (UniqueName: \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\") pod \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") "
Oct 31 03:53:20.085811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.085769    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\") pod \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") "
Oct 31 03:53:20.086220 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.085920    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\") pod \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\" (UID: \"953bf90c-14d4-4c33-8d48-dfc8bf3351c6\") "
Oct 31 03:53:20.086858 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.086823    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca" pod "953bf90c-14d4-4c33-8d48-dfc8bf3351c6" (UID: "953bf90c-14d4-4c33-8d48-dfc8bf3351c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:20.087289 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.087257    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.095154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.094065    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr" pod "953bf90c-14d4-4c33-8d48-dfc8bf3351c6" (UID: "953bf90c-14d4-4c33-8d48-dfc8bf3351c6"). InnerVolumeSpecName "kube-api-access-nwpfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:20.095154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.094150    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca" pod "953bf90c-14d4-4c33-8d48-dfc8bf3351c6" (UID: "953bf90c-14d4-4c33-8d48-dfc8bf3351c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:53:20.189098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.189061    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpfr\" (UniqueName: \"kubernetes.io/projected/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-kube-api-access-nwpfr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.189098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.189101    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953bf90c-14d4-4c33-8d48-dfc8bf3351c6-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.828711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.828680    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"
Oct 31 03:53:20.828906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.828722    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6" event={"ID":"953bf90c-14d4-4c33-8d48-dfc8bf3351c6","Type":"ContainerDied","Data":"d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30"}
Oct 31 03:53:20.828906 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:20.828782    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d305849994eaf8f485c3043284a6fb53494f8abe2808bfd4aed32fc75239ff30"
Oct 31 03:53:23.048657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:23.048616    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"]
Oct 31 03:53:23.093213 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:23.093182    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073e8d96-s4cnl"]
Oct 31 03:53:24.136029 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:24.135996    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ce95fe-a438-492b-b8b7-be170966f13b" path="/var/lib/kubelet/pods/d8ce95fe-a438-492b-b8b7-be170966f13b/volumes"
Oct 31 03:53:29.456058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.456025    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 03:53:29.456862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.456842    5831 cpu_manager.go:414] "containerMap: removing container" podUID="953bf90c-14d4-4c33-8d48-dfc8bf3351c6" containerName="revoke-keys"
Oct 31 03:53:29.456937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.456864    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953bf90c-14d4-4c33-8d48-dfc8bf3351c6" containerName="revoke-keys"
Oct 31 03:53:29.457018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.457007    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="953bf90c-14d4-4c33-8d48-dfc8bf3351c6" containerName="revoke-keys"
Oct 31 03:53:29.462296 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.462276    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521174    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521229    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dnq\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521460 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521261    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521482    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521646 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521596    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521696    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521779 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521731    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521831    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.521904 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.521895    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.528589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.528562    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-dffe64-openshift-default-dockercfg-zdb5g\""
Oct 31 03:53:29.561770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.561729    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 03:53:29.702583 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.702550    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.703589 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.703562    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dnq\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.704243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.704221    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.705446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.705426    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.705971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.705953    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.706466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.706414    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.734811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.734779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.735503 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.735481    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.736482 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.736460    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.736706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.709980    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.736839 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.705879    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.736946 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.736323    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.737045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.704652    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.737141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.703491    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.799558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.798999    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.803557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.802347    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.808970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.808706    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dnq\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:29.811932 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:29.811907    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\") pod \"e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") " pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:30.077472 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.077387    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:30.522921 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.522892    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 03:53:30.523933 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:53:30.523901    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38e149c_aca7_43d5_a10d_b5a41216b9ba.slice/crio-cfc91a7d81d62eabf4571506c6f2e986cb387edea05d641ac46e2b8df651152c WatchSource:0}: Error finding container cfc91a7d81d62eabf4571506c6f2e986cb387edea05d641ac46e2b8df651152c: Status 404 returned error can't find the container with id cfc91a7d81d62eabf4571506c6f2e986cb387edea05d641ac46e2b8df651152c
Oct 31 03:53:30.526996 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.526961    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:30.527076 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.527047    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:30.527116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.527088    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:30.916485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.916383    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerStarted","Data":"0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03"}
Oct 31 03:53:30.916485 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:30.916441    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerStarted","Data":"cfc91a7d81d62eabf4571506c6f2e986cb387edea05d641ac46e2b8df651152c"}
Oct 31 03:53:31.077609 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:31.077572    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:31.083711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:31.083669    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:31.117420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:31.117369    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podStartSLOduration=2.11735577 podStartE2EDuration="2.11735577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:31.071363123 +0000 UTC m=+2761.085924843" watchObservedRunningTime="2026-10-31 03:53:31.11735577 +0000 UTC m=+2761.131917478"
Oct 31 03:53:31.925184 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:31.925153    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:31.926362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:31.926342    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:53:35.547862 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:35.547822    5831 scope.go:122] "RemoveContainer" containerID="13daff079fc0bef4b8f264782e111f6654db75865f7683b298c8a87b919789a2"
Oct 31 03:53:35.564917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:35.564875    5831 scope.go:122] "RemoveContainer" containerID="4d8c627d05a28cbd5477397b9781c7d3285293efc8a884175c2d49e3a84d2a4f"
Oct 31 03:53:40.113553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.107136    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"]
Oct 31 03:53:40.218191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.218135    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"]
Oct 31 03:53:40.218191 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.218190    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"]
Oct 31 03:53:40.224877 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.224840    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3fd9b5b-b56mn"]
Oct 31 03:53:40.231610 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.231579    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:53:40.239742 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.239713    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.247019 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.246987    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:53:40.262118 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:53:40.262079    5831 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-0509fc0b-q62x6\" is forbidden: User \"system:node:ip-10-0-140-194.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-194.us-west-2.compute.internal' and this object" podUID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0509fc0b-q62x6"
Oct 31 03:53:40.288813 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.288784    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dffe64\""
Oct 31 03:53:40.288984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.288836    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dffe64-dockercfg-hzl5p\""
Oct 31 03:53:40.311726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.311685    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.311895 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.311860    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfgq\" (UniqueName: \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.412757 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.412655    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.412963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.412794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfgq\" (UniqueName: \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.413380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.413358    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.466300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.466270    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfgq\" (UniqueName: \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\") pod \"payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") " pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.557780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.557736    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:40.944356 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:53:40.944325    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bedd4ef_617f_4a03_b0f0_51ee6a6c2361.slice/crio-ee379d7556046f613a3d929e509ae4968f0d650f7eeb22a6fa47ec39281edbae WatchSource:0}: Error finding container ee379d7556046f613a3d929e509ae4968f0d650f7eeb22a6fa47ec39281edbae: Status 404 returned error can't find the container with id ee379d7556046f613a3d929e509ae4968f0d650f7eeb22a6fa47ec39281edbae
Oct 31 03:53:40.944557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:40.944395    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:53:41.179119 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:41.179074    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" event={"ID":"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361","Type":"ContainerStarted","Data":"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"}
Oct 31 03:53:41.179662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:41.179144    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" event={"ID":"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361","Type":"ContainerStarted","Data":"ee379d7556046f613a3d929e509ae4968f0d650f7eeb22a6fa47ec39281edbae"}
Oct 31 03:53:41.179662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:41.179171    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:41.287202 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:41.287064    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" podStartSLOduration=1.287046332 podStartE2EDuration="1.287046332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:41.286670836 +0000 UTC m=+2771.301232545" watchObservedRunningTime="2026-10-31 03:53:41.287046332 +0000 UTC m=+2771.301608055"
Oct 31 03:53:41.947156 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:41.947122    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:53:42.141173 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:42.141139    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7ac748-67f4-49fd-95b6-3046ba44cd1f" path="/var/lib/kubelet/pods/6d7ac748-67f4-49fd-95b6-3046ba44cd1f/volumes"
Oct 31 03:53:42.141734 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:42.141718    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd80bf2-1ae0-4b14-9873-c9cc15d1255e" path="/var/lib/kubelet/pods/dfd80bf2-1ae0-4b14-9873-c9cc15d1255e/volumes"
Oct 31 03:53:42.979942 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:53:42.979908    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:53:52.194521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:52.194487    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:53:53.389133 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.389097    5831 generic.go:356] "Generic (PLEG): container finished" podID="db69b913-b751-4968-b216-4358ea07a83f" containerID="4c6aaea2480fb21d3f224f9046cb49ee5ff00f4a3c3be51e6677e19b1019e5f2" exitCode=0
Oct 31 03:53:53.389700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.389184    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerDied","Data":"4c6aaea2480fb21d3f224f9046cb49ee5ff00f4a3c3be51e6677e19b1019e5f2"}
Oct 31 03:53:53.389700 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.389622    5831 scope.go:122] "RemoveContainer" containerID="4c6aaea2480fb21d3f224f9046cb49ee5ff00f4a3c3be51e6677e19b1019e5f2"
Oct 31 03:53:53.391237 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.391210    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:53.391317 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.391292    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:53.391361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:53.391324    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:53:54.413045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:54.412999    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerStarted","Data":"49fa6258b5acd92249d9cc2fdd0a421b75f66051c31fa94bd6bcfe137c8f3014"}
Oct 31 03:53:55.111911 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:55.111874    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:53:55.112134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:55.112086    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:53:55.112207 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:55.112130    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:53:56.112986 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:56.112949    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:53:56.113453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:56.113014    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:53:57.048659 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:57.048625    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"]
Oct 31 03:53:57.110361 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:57.110326    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f222b97-hzm89"]
Oct 31 03:53:57.113026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:57.112993    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:53:57.113392 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:57.113054    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:53:58.112457 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:58.112416    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:53:58.112669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:58.112477    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:53:58.136351 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:58.136319    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f559b11-bbbb-4ef5-9947-114c9476384e" path="/var/lib/kubelet/pods/6f559b11-bbbb-4ef5-9947-114c9476384e/volumes"
Oct 31 03:53:59.112379 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:59.112337    5831 patch_prober.go:29] interesting pod/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 31 03:53:59.112625 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:53:59.112407    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 31 03:54:00.118656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:00.118618    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:54:00.119198 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:00.119166    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:54:00.119952 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:00.119933    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 03:54:07.170830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.170794    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 03:54:07.176634 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.176612    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.246632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.246243    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 03:54:07.246632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.246381    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5kb\" (UniqueName: \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.246632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.246482    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.246632 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.246593    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.246963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.246714    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.269944 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.269910    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a5f70c-serving-cert\""
Oct 31 03:54:07.348403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.348364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5kb\" (UniqueName: \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.348647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.348471    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.348647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.348541    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.348647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.348579    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.348647 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:54:07.348639    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-serving-cert: secret "maas-api-e2e-shared-b-w1-a5f70c-serving-cert" not found
Oct 31 03:54:07.349059 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:54:07.348727    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls podName:8b55214a-a131-4c5d-bfc8-aaa2553560f8 nodeName:}" failed. No retries permitted until 2026-10-31 03:54:07.848704414 +0000 UTC m=+2797.863266115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls") pod "maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" (UID: "8b55214a-a131-4c5d-bfc8-aaa2553560f8") : secret "maas-api-e2e-shared-b-w1-a5f70c-serving-cert" not found
Oct 31 03:54:07.349479 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.349458    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.358569 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.352128    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.448030 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.447995    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5kb\" (UniqueName: \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.902231 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.902122    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:07.935523 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:07.935487    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:08.112222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:08.111912    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:08.575829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:08.575792    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 03:54:08.638574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:08.638016    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:54:08.641719 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:54:08.641674    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b55214a_a131_4c5d_bfc8_aaa2553560f8.slice/crio-9b7d4123173ef34f92b5e4ce603a457cceee4536bb7cd359161c5aedde2a2e28 WatchSource:0}: Error finding container 9b7d4123173ef34f92b5e4ce603a457cceee4536bb7cd359161c5aedde2a2e28: Status 404 returned error can't find the container with id 9b7d4123173ef34f92b5e4ce603a457cceee4536bb7cd359161c5aedde2a2e28
Oct 31 03:54:08.674254 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:08.674224    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" event={"ID":"8b55214a-a131-4c5d-bfc8-aaa2553560f8","Type":"ContainerStarted","Data":"9b7d4123173ef34f92b5e4ce603a457cceee4536bb7cd359161c5aedde2a2e28"}
Oct 31 03:54:10.047547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:10.047493    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"]
Oct 31 03:54:10.119553 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:10.119344    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5a5fd3-n54mn"]
Oct 31 03:54:10.146074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:10.146038    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4" path="/var/lib/kubelet/pods/5b9996bd-c4f0-4ee7-acf1-8e2f0a593eb4/volumes"
Oct 31 03:54:10.757973 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:10.757939    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" event={"ID":"8b55214a-a131-4c5d-bfc8-aaa2553560f8","Type":"ContainerStarted","Data":"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"}
Oct 31 03:54:10.868424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:10.868358    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" podStartSLOduration=2.935442122 podStartE2EDuration="3.868338981s" totalImagesPullingTime="932.896859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:10.865899944 +0000 UTC m=+2800.880461653" watchObservedRunningTime="2026-10-31 03:54:10.868338981 +0000 UTC m=+2800.882900690"
Oct 31 03:54:11.766006 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:11.765971    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:12.983826 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:54:12.983790    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:54:16.490686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.490645    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:54:16.503058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.490726    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:54:16.818884 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.818856    5831 generic.go:356] "Generic (PLEG): container finished" podID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerID="77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de" exitCode=0
Oct 31 03:54:16.819043 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.818940    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerDied","Data":"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"}
Oct 31 03:54:16.819339 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.819325    5831 scope.go:122] "RemoveContainer" containerID="77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"
Oct 31 03:54:16.820977 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.820946    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:16.821105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.821034    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:16.821105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:16.821096    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:17.813723 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.813682    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 03:54:17.831484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.831448    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerStarted","Data":"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"}
Oct 31 03:54:17.833337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.833307    5831 generic.go:356] "Generic (PLEG): container finished" podID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerID="3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64" exitCode=0
Oct 31 03:54:17.833337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.833327    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerDied","Data":"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"}
Oct 31 03:54:17.833736 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.833721    5831 scope.go:122] "RemoveContainer" containerID="3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"
Oct 31 03:54:17.835193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.835164    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:17.835309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.835230    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:17.835309 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.835262    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:17.986745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.986713    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:54:17.987653 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.987079    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:17.987864 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:17.987837    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:18.843515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:18.843477    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerStarted","Data":"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"}
Oct 31 03:54:18.992556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:18.988988    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:18.992556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:18.989057    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:19.341565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:19.341377    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:54:19.341565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:19.341506    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:19.341876 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:19.341578    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:20.000556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:19.991971    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:20.000556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:19.992063    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:20.343673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:20.342984    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:20.343673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:20.343046    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:20.990556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:20.986917    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:20.990556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:20.986993    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:21.343059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:21.342939    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:21.343059 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:21.343013    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:21.992557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:21.988955    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:21.992557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:21.989018    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:22.342036 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:22.341937    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:22.344114 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:22.344075    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:22.994146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:22.989979    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:22.994146 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:22.990045    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:23.341671 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:23.341577    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:23.341947 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:23.341919    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:23.993418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:23.993373    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 31 03:54:23.994102 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:23.994064    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 31 03:54:24.118823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:24.118780    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:54:24.342180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:24.342093    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 31 03:54:24.342180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:24.342157    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 31 03:54:24.994399 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:24.994366    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:54:24.995930 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:24.995902    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 03:54:25.046105 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:25.046072    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"]
Oct 31 03:54:25.095054 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:25.095024    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee846898-wnjvn"]
Oct 31 03:54:25.345442 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:25.345361    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:54:25.345848 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:25.345831    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:54:25.346624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:25.346601    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 03:54:26.137179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:26.137148    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4227d42-b4a7-4e92-9100-dc339a18c62e" path="/var/lib/kubelet/pods/a4227d42-b4a7-4e92-9100-dc339a18c62e/volumes"
Oct 31 03:54:26.151598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:26.151571    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:54:28.657729 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.657695    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 03:54:28.662887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.662867    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711314    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711432    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvm2j\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711471    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711511    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711577    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711611    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711691    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711719    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.714579 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.711754    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.719970 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.719937    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8b85a6-openshift-default-dockercfg-pq8bg\""
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827072    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvm2j\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827140    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827197    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827260    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827300    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827417    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827456    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.827572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.827502    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.830711 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.828368    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.830985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.830905    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.831641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.831512    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.832608 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.832589    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.834425 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.834398    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.834831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.834812    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.906558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.896097    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.930663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.929944    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.930663 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.930382    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvm2j\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:28.994390 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:28.952681    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 03:54:29.007956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.007879    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\") pod \"e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") " pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:29.277917 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.277836    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:29.717100 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.717066    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 03:54:29.718107 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:54:29.718076    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e65b3a_a1fd_40bc_be30_8f1d2830d9bc.slice/crio-2873b331e1e0352b41828bfe55cd0f22ce99f4e843ffea14b792a872bae40125 WatchSource:0}: Error finding container 2873b331e1e0352b41828bfe55cd0f22ce99f4e843ffea14b792a872bae40125: Status 404 returned error can't find the container with id 2873b331e1e0352b41828bfe55cd0f22ce99f4e843ffea14b792a872bae40125
Oct 31 03:54:29.720760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.720728    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:29.720847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.720801    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:29.720847 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:29.720837    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:54:30.028235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:30.028136    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerStarted","Data":"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"}
Oct 31 03:54:30.028235 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:30.028205    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerStarted","Data":"2873b331e1e0352b41828bfe55cd0f22ce99f4e843ffea14b792a872bae40125"}
Oct 31 03:54:30.141874 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:30.141813    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podStartSLOduration=2.141795718 podStartE2EDuration="2.141795718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:30.137335431 +0000 UTC m=+2820.151897140" watchObservedRunningTime="2026-10-31 03:54:30.141795718 +0000 UTC m=+2820.156357427"
Oct 31 03:54:30.283631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:30.278856    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:30.306566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:30.303958    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:31.037083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:31.037050    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:31.038117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:31.038094    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:54:35.628160 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.628122    5831 scope.go:122] "RemoveContainer" containerID="15d35375f26f7dc4b0fdf7e9daf541abe2852298208f02ad8ff04372a7e83392"
Oct 31 03:54:35.640072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.640050    5831 scope.go:122] "RemoveContainer" containerID="484d75138df3be136c7de7c859a3aca0b2f6f1ed9c0a7f70b38eb831b25d77de"
Oct 31 03:54:35.650342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.650323    5831 scope.go:122] "RemoveContainer" containerID="305d065e6754f10681456bc2153b6da8a2f6c18da11ab81853b69017ddfa825c"
Oct 31 03:54:35.659922 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.659900    5831 scope.go:122] "RemoveContainer" containerID="85ab4c65670b7b6e47204bf28f2d527c9eaa60faf494509be558ddcff8610d66"
Oct 31 03:54:35.670504 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.670485    5831 scope.go:122] "RemoveContainer" containerID="726241737eb81a1fe3dab5583816e32429ed8f9c0bc33f2104a90bdc0de7cf83"
Oct 31 03:54:35.681429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:35.681405    5831 scope.go:122] "RemoveContainer" containerID="f361df647ca83dab691a46dbd55168f56918d78a50dea93fa58d4fdff8f05d88"
Oct 31 03:54:39.045418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:39.045374    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"]
Oct 31 03:54:39.091064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:39.091029    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf83654-ws9g8"]
Oct 31 03:54:40.151183 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:40.151140    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2426bd03-8970-451b-846d-5f5813e737e2" path="/var/lib/kubelet/pods/2426bd03-8970-451b-846d-5f5813e737e2/volumes"
Oct 31 03:54:42.980401 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:54:42.980360    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:54:49.045875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:49.045839    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"]
Oct 31 03:54:49.091811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:49.091778    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425dc93c-v94r2"]
Oct 31 03:54:50.137951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:50.137914    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cac73b5-4073-48b9-a4e8-6d12264ca322" path="/var/lib/kubelet/pods/6cac73b5-4073-48b9-a4e8-6d12264ca322/volumes"
Oct 31 03:54:54.045301 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:54.045264    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"]
Oct 31 03:54:54.093455 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:54.093418    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92315a5a-lnplc"]
Oct 31 03:54:54.137435 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:54:54.137401    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b71738-401f-428e-b569-29f5f5fbedcf" path="/var/lib/kubelet/pods/b6b71738-401f-428e-b569-29f5f5fbedcf/volumes"
Oct 31 03:55:04.062186 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:04.062147    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"]
Oct 31 03:55:04.115935 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:04.115887    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0229bc26-l4qq4"]
Oct 31 03:55:04.136846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:04.136800    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886bd70c-8d32-4614-96e4-ee3ab3592bcc" path="/var/lib/kubelet/pods/886bd70c-8d32-4614-96e4-ee3ab3592bcc/volumes"
Oct 31 03:55:09.044462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.044425    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"]
Oct 31 03:55:09.090581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.090477    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f3b76e9-kmpst"]
Oct 31 03:55:09.506956 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.506920    5831 generic.go:356] "Generic (PLEG): container finished" podID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerID="0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03" exitCode=0
Oct 31 03:55:09.507140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.507002    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerDied","Data":"0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03"}
Oct 31 03:55:09.507571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.507554    5831 scope.go:122] "RemoveContainer" containerID="0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03"
Oct 31 03:55:09.509422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.509391    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:09.509592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.509493    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:09.509592 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:09.509563    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:10.139924 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:10.139889    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9" path="/var/lib/kubelet/pods/aa54cc50-4f91-4262-8bc4-5fdf0cda9eb9/volumes"
Oct 31 03:55:10.517411 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:10.517371    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerStarted","Data":"5730ca211f6a81bda97794ec7272a628794be01d1e66948608394fc739c69801"}
Oct 31 03:55:11.078420 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:11.078387    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:55:11.078683 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:11.078628    5831 patch_prober.go:29] interesting pod/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 31 03:55:11.078770 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:11.078678    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 31 03:55:12.089965 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:12.089925    5831 patch_prober.go:29] interesting pod/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 31 03:55:12.090614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:12.090583    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 31 03:55:12.993565 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:55:12.993234    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:55:13.084464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:13.083649    5831 patch_prober.go:29] interesting pod/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 31 03:55:13.084464 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:13.083731    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 31 03:55:14.078226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:14.078193    5831 patch_prober.go:29] interesting pod/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 31 03:55:14.078703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:14.078272    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 31 03:55:15.046767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:15.046726    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"]
Oct 31 03:55:15.078481 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:15.078451    5831 patch_prober.go:29] interesting pod/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 31 03:55:15.079040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:15.078570    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 31 03:55:15.103555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:15.101466    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79be76-bh9wr"]
Oct 31 03:55:16.083412 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:16.083383    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:55:16.083931 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:16.083707    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:55:16.084652 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:16.084630    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:55:16.136772 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:16.136740    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c124b6d3-9b28-4230-ba43-3f60ab4eb215" path="/var/lib/kubelet/pods/c124b6d3-9b28-4230-ba43-3f60ab4eb215/volumes"
Oct 31 03:55:21.045367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:21.045328    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"]
Oct 31 03:55:21.091022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:21.090984    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c7b27b-gbf2r"]
Oct 31 03:55:22.141343 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:22.141303    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5072f4c6-0316-485a-80fd-ba2d97f08dcc" path="/var/lib/kubelet/pods/5072f4c6-0316-485a-80fd-ba2d97f08dcc/volumes"
Oct 31 03:55:26.044048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:26.044009    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"]
Oct 31 03:55:26.090052 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:26.090015    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893e053b-mzlpq"]
Oct 31 03:55:26.136547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:26.136496    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ecec33-e33a-40fb-a818-ec5368293613" path="/var/lib/kubelet/pods/71ecec33-e33a-40fb-a818-ec5368293613/volumes"
Oct 31 03:55:27.543817 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:27.521608    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:55:31.909103 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:31.908998    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:55:31.909623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:31.909210    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy" containerID="cri-o://4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d" gracePeriod=30
Oct 31 03:55:33.654765 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:33.654717    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:55:33.655236 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:33.654794    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:55:35.753099 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.753062    5831 scope.go:122] "RemoveContainer" containerID="171eaecc3e76efbab95b01c34257d59781f104931e334bbd1759f11b3470da08"
Oct 31 03:55:35.765329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.765299    5831 scope.go:122] "RemoveContainer" containerID="5e0bc80d7dc47ff753f24f89691ced7a9081e6a2231a66491dc58b42f1c7b4e4"
Oct 31 03:55:35.776424 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.776392    5831 scope.go:122] "RemoveContainer" containerID="e1a40132a55502b8cc7d82141958b48443c1b9ae3c53a8388fbfbf85fd7946ab"
Oct 31 03:55:35.789674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.789654    5831 scope.go:122] "RemoveContainer" containerID="cee54058afdcdb54cdc87a41eb815258b583ea6fed4521b1fd7a2ab08a768a7b"
Oct 31 03:55:35.804709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.804684    5831 scope.go:122] "RemoveContainer" containerID="2d4822cd1e905504fcf486f4bd92c4ecdc3e503e13a05006d48fb61023f64ddb"
Oct 31 03:55:35.816312 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:35.816294    5831 scope.go:122] "RemoveContainer" containerID="c0a8ef76988fcc659c176079bba81d862d535177b60b252973237454d1b98b68"
Oct 31 03:55:36.851244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.851210    5831 generic.go:356] "Generic (PLEG): container finished" podID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerID="2af151d3490945def1a9380bfd00fc46c874e58c4c1dea2e9775799500dfb6d4" exitCode=0
Oct 31 03:55:36.851860 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.851285    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerDied","Data":"2af151d3490945def1a9380bfd00fc46c874e58c4c1dea2e9775799500dfb6d4"}
Oct 31 03:55:36.851943 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.851861    5831 scope.go:122] "RemoveContainer" containerID="2af151d3490945def1a9380bfd00fc46c874e58c4c1dea2e9775799500dfb6d4"
Oct 31 03:55:36.853576 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.853550    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:36.853662 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.853644    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:36.853702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:36.853676    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:55:37.270570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.268448    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:55:37.284649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284621    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.284802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284696    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.284802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284744    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.284913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284826    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.284913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284874    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.284913 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284902    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.285086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284957    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.285086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.284995    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.285086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285060    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\") pod \"58de3d62-c934-41d0-91c6-7c9b18129df0\" (UID: \"58de3d62-c934-41d0-91c6-7c9b18129df0\") "
Oct 31 03:55:37.285086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285063    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:37.285333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285195    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:37.285428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285402    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:37.285837 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285615    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:37.285994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285891    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.285994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285914    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58de3d62-c934-41d0-91c6-7c9b18129df0-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.285994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285928    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.285994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.285942    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.286245 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.286006    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:37.287669 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.287641    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "kube-api-access-nkmw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:37.287811 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.287796    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:37.287985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.287962    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:37.288660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.288635    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo" pod "58de3d62-c934-41d0-91c6-7c9b18129df0" (UID: "58de3d62-c934-41d0-91c6-7c9b18129df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:37.388140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.388053    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.388140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.388091    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.388140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.388108    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.388140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.388124    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkmw8\" (UniqueName: \"kubernetes.io/projected/58de3d62-c934-41d0-91c6-7c9b18129df0-kube-api-access-nkmw8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.388140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.388139    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58de3d62-c934-41d0-91c6-7c9b18129df0-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:37.861983 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.861938    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerStarted","Data":"ca7f0438d95ffcad42b193a570f245cd02ec82361ba47bee62cf8512c5336598"}
Oct 31 03:55:37.863537 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.863503    5831 generic.go:356] "Generic (PLEG): container finished" podID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerID="4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d" exitCode=0
Oct 31 03:55:37.863684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.863561    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"
Oct 31 03:55:37.863684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.863587    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" event={"ID":"58de3d62-c934-41d0-91c6-7c9b18129df0","Type":"ContainerDied","Data":"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"}
Oct 31 03:55:37.863684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.863637    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc" event={"ID":"58de3d62-c934-41d0-91c6-7c9b18129df0","Type":"ContainerDied","Data":"51285f7312950c02df0a6454e64a6c297371ec7470f6bf2c4fe97f1ba2d47eb1"}
Oct 31 03:55:37.863684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.863661    5831 scope.go:122] "RemoveContainer" containerID="4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"
Oct 31 03:55:37.910557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.906950    5831 scope.go:122] "RemoveContainer" containerID="4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"
Oct 31 03:55:37.910557 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:55:37.907396    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d\": container with ID starting with 4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d not found: ID does not exist" containerID="4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"
Oct 31 03:55:37.910557 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:37.907431    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d"} err="failed to get container status \"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d\": rpc error: code = NotFound desc = could not find container \"4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d\": container with ID starting with 4a4069c79458a3036db20bc33cc9f1ea673019652c8d58c7a8c6f2d0e6cdc31d not found: ID does not exist"
Oct 31 03:55:38.110791 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.110717    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:55:38.124937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.119437    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-jh7tc"]
Oct 31 03:55:38.135849 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.135817    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" path="/var/lib/kubelet/pods/58de3d62-c934-41d0-91c6-7c9b18129df0/volumes"
Oct 31 03:55:38.292556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.288950    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:55:38.292556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.289011    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 31 03:55:38.292556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:38.289082    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 31 03:55:39.302693 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:39.302652    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 31 03:55:39.303367 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:39.303331    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 31 03:55:40.294556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:40.290896    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 31 03:55:40.294556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:40.290976    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 31 03:55:41.296556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:41.295909    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 31 03:55:41.296556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:41.295974    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 31 03:55:42.288937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:42.288899    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 31 03:55:42.289258 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:42.289231    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 31 03:55:43.016617 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:55:43.016582    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:55:43.293654 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:43.293574    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:55:43.293994 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:43.293970    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:55:43.294660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:43.294639    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 03:55:44.045098 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:44.045010    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"]
Oct 31 03:55:44.048726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:44.048702    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba679f60-qjhv2"]
Oct 31 03:55:44.136918 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:44.136882    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f3676d-7652-4968-a3cd-aed5de7f022f" path="/var/lib/kubelet/pods/06f3676d-7652-4968-a3cd-aed5de7f022f/volumes"
Oct 31 03:55:47.405397 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.405360    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 03:55:47.406316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.406292    5831 cpu_manager.go:414] "containerMap: removing container" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy"
Oct 31 03:55:47.406316 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.406318    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy"
Oct 31 03:55:47.406571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.406504    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="58de3d62-c934-41d0-91c6-7c9b18129df0" containerName="istio-proxy"
Oct 31 03:55:47.410117 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.410096    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.462447 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.462421    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 03:55:47.503774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.503737    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-8b85a6-serving-cert\""
Oct 31 03:55:47.542509 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.542475    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.542676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.542516    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.542676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.542638    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ctx\" (UniqueName: \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.542774 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.542749    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.644270 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.644234    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.644459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.644282    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.644459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.644322    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ctx\" (UniqueName: \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.644459 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.644402    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.644640 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:55:47.644561    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-serving-cert: secret "maas-api-e2e-praxis-b-8b85a6-serving-cert" not found
Oct 31 03:55:47.644710 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:55:47.644651    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls podName:3050f374-34af-4dc7-8659-1eeb998480b2 nodeName:}" failed. No retries permitted until 2026-10-31 03:55:48.14462906 +0000 UTC m=+2898.159190764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls") pod "maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" (UID: "3050f374-34af-4dc7-8659-1eeb998480b2") : secret "maas-api-e2e-praxis-b-8b85a6-serving-cert" not found
Oct 31 03:55:47.645111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.645079    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.647342 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.647310    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:47.716120 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:47.706809    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ctx\" (UniqueName: \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:48.152109 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:48.152009    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:48.154802 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:48.154764    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:48.326995 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:48.326965    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:48.702275 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:48.702247    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 03:55:48.703947 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:55:48.703896    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3050f374_34af_4dc7_8659_1eeb998480b2.slice/crio-a33f25627f2476f68f627579d040f86902c7a7ad484855cbd19c6094529fc335 WatchSource:0}: Error finding container a33f25627f2476f68f627579d040f86902c7a7ad484855cbd19c6094529fc335: Status 404 returned error can't find the container with id a33f25627f2476f68f627579d040f86902c7a7ad484855cbd19c6094529fc335
Oct 31 03:55:49.027429 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:49.027343    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" event={"ID":"3050f374-34af-4dc7-8659-1eeb998480b2","Type":"ContainerStarted","Data":"a33f25627f2476f68f627579d040f86902c7a7ad484855cbd19c6094529fc335"}
Oct 31 03:55:50.037291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:50.037188    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" event={"ID":"3050f374-34af-4dc7-8659-1eeb998480b2","Type":"ContainerStarted","Data":"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"}
Oct 31 03:55:50.037291 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:50.037273    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:55:50.145246 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:50.145187    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" podStartSLOduration=2.12896335 podStartE2EDuration="3.145173382s" totalImagesPullingTime="1.016210032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:50.143015834 +0000 UTC m=+2900.157577541" watchObservedRunningTime="2026-10-31 03:55:50.145173382 +0000 UTC m=+2900.159735090"
Oct 31 03:55:56.050378 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:55:56.050346    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 03:56:01.716253 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:01.716211    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:56:03.925185 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:03.925151    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 03:56:03.930089 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:03.930066    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.009398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.009359    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 03:56:04.023891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.023854    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-93f411\"/\"openshift-service-ca.crt\""
Oct 31 03:56:04.023891 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.023869    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-93f411\"/\"kube-root-ca.crt\""
Oct 31 03:56:04.024162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.023895    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-93f411\"/\"default-dockercfg-8gms5\""
Oct 31 03:56:04.024162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.024038    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-93f411\"/\"auth-test-model-93f411-kserve-self-signed-certs\""
Oct 31 03:56:04.026428 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026398    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxml\" (UniqueName: \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.026581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026460    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.026581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026569    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.026686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026631    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.026751 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026720    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.026901 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.026887    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130430    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130499    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130773    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130861    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxml\" (UniqueName: \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130903    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.131086 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.130944    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.132136 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.132103    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.133014 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.132990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.133446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.133424    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.142692 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.141894    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.153027 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.152993    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.219893 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.219859    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxml\" (UniqueName: \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\") pod \"auth-test-model-93f411-kserve-57d878f445-h5mdv\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") " pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.243434 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.243410    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:04.652377 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:04.652289    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 03:56:04.656380 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:56:04.656342    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b20ebe2_66aa_409e_aff3_99d63cf54f4e.slice/crio-294601473c9835753ca78f82a993540c363dc84d81fa4a2818165e6303d567b0 WatchSource:0}: Error finding container 294601473c9835753ca78f82a993540c363dc84d81fa4a2818165e6303d567b0: Status 404 returned error can't find the container with id 294601473c9835753ca78f82a993540c363dc84d81fa4a2818165e6303d567b0
Oct 31 03:56:05.231782 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:05.231745    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" event={"ID":"8b20ebe2-66aa-409e-aff3-99d63cf54f4e","Type":"ContainerStarted","Data":"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"}
Oct 31 03:56:05.232311 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:05.231801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" event={"ID":"8b20ebe2-66aa-409e-aff3-99d63cf54f4e","Type":"ContainerStarted","Data":"294601473c9835753ca78f82a993540c363dc84d81fa4a2818165e6303d567b0"}
Oct 31 03:56:05.243823 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:05.243798    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:05.358702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:05.358640    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" podStartSLOduration=2.358624762 podStartE2EDuration="2.358624762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:05.356221311 +0000 UTC m=+2915.370783041" watchObservedRunningTime="2026-10-31 03:56:05.358624762 +0000 UTC m=+2915.373186469"
Oct 31 03:56:12.979344 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:12.979299    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:56:13.366715 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.366636    5831 generic.go:356] "Generic (PLEG): container finished" podID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerID="eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397" exitCode=0
Oct 31 03:56:13.366872 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.366707    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerDied","Data":"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"}
Oct 31 03:56:13.367193 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.367178    5831 scope.go:122] "RemoveContainer" containerID="eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"
Oct 31 03:56:13.368745 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.368719    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:56:13.368829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.368789    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:56:13.368829 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:13.368823    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 03:56:14.244241 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:14.244202    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:14.257329 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:14.257304    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:14.376465 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:14.376432    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerStarted","Data":"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"}
Oct 31 03:56:14.394344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:14.394261    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 03:56:15.284555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:15.284312    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:56:15.284555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:15.284444    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 31 03:56:15.284555 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:15.284488    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 31 03:56:16.037433 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:16.037393    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:56:16.287615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:16.282884    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 31 03:56:16.287615 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:16.282942    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 31 03:56:17.302556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:17.301952    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 31 03:56:17.302556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:17.302027    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 31 03:56:18.280155 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:18.280118    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 31 03:56:18.280468 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:18.280440    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 31 03:56:19.278271 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:19.278241    5831 patch_prober.go:29] interesting pod/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 31 03:56:19.278760 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:19.278289    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 31 03:56:20.284565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:20.284506    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:56:20.286022 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:20.285995    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 03:56:29.036440 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.036396    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:56:29.037015 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.036681    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy" containerID="cri-o://2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92" gracePeriod=30
Oct 31 03:56:29.083512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.083475    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:56:29.083806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.083771    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy" containerID="cri-o://18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d" gracePeriod=30
Oct 31 03:56:29.089058 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.089028    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:56:29.089281 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.089256    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" containerID="cri-o://7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f" gracePeriod=30
Oct 31 03:56:29.131998 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.131964    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:56:29.132244 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:29.132196    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy" containerID="cri-o://7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9" gracePeriod=30
Oct 31 03:56:34.441046 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.441018    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:56:34.511603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.509857    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511399    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511515    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511636    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511767    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511799    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.511888 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511841    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.512111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511888    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.512111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.510408    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:56:34.512111 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.511929    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\") pod \"c686c0cc-a050-45dc-be8f-e934579c4db0\" (UID: \"c686c0cc-a050-45dc-be8f-e934579c4db0\") "
Oct 31 03:56:34.512261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512210    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.512694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512425    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.512694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512666    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.512694 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512688    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.512945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512723    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.512945 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.512852    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.513512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.513488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.516304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.516268    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:34.519081 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.519040    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "kube-api-access-88jsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.519704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.519671    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.540648 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.540604    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy" pod "c686c0cc-a050-45dc-be8f-e934579c4db0" (UID: "c686c0cc-a050-45dc-be8f-e934579c4db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.575624 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.575595    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:56:34.578584 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.578558    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:56:34.580514 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.580486    5831 generic.go:356] "Generic (PLEG): container finished" podID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerID="7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9" exitCode=0
Oct 31 03:56:34.580657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.580560    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" event={"ID":"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7","Type":"ContainerDied","Data":"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"}
Oct 31 03:56:34.580657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.580577    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"
Oct 31 03:56:34.580657 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.580641    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77" event={"ID":"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7","Type":"ContainerDied","Data":"e4b230e95a4e039855c76d971347f95d6a5a2e49bfc2c7256020e14a99917863"}
Oct 31 03:56:34.580831 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.580676    5831 scope.go:122] "RemoveContainer" containerID="7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"
Oct 31 03:56:34.582598 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.582573    5831 generic.go:356] "Generic (PLEG): container finished" podID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerID="18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d" exitCode=0
Oct 31 03:56:34.582702 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.582667    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" event={"ID":"e8bb4852-ad0f-498c-9597-e868ba9556dc","Type":"ContainerDied","Data":"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"}
Oct 31 03:56:34.582755 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.582710    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m" event={"ID":"e8bb4852-ad0f-498c-9597-e868ba9556dc","Type":"ContainerDied","Data":"28d385b3984016be8d5f6b5a79f88e9253f37d7a36dacdbb320bfbcf4c7e900f"}
Oct 31 03:56:34.582796 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.582778    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"
Oct 31 03:56:34.584484 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.584460    5831 generic.go:356] "Generic (PLEG): container finished" podID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerID="7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f" exitCode=0
Oct 31 03:56:34.584664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.584499    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"
Oct 31 03:56:34.584664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.584558    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" event={"ID":"e84b7ce9-cced-420b-89f6-b897f79430a0","Type":"ContainerDied","Data":"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"}
Oct 31 03:56:34.584664 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.584614    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" event={"ID":"e84b7ce9-cced-420b-89f6-b897f79430a0","Type":"ContainerDied","Data":"8a3b18abec10dcb845b95379bda4a9983575b2a0fc98cf3f78bebe9cb86a70b0"}
Oct 31 03:56:34.586674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.586653    5831 generic.go:356] "Generic (PLEG): container finished" podID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerID="2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92" exitCode=0
Oct 31 03:56:34.586780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.586711    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerDied","Data":"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"}
Oct 31 03:56:34.586780 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.586749    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb" event={"ID":"c686c0cc-a050-45dc-be8f-e934579c4db0","Type":"ContainerDied","Data":"792764ee9a73eb5afbae06af31f799a74619450095ddff900fd8becbb31ef39e"}
Oct 31 03:56:34.586915 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.586819    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"
Oct 31 03:56:34.608292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.599695    5831 scope.go:122] "RemoveContainer" containerID="7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"
Oct 31 03:56:34.608292 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:34.600035    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9\": container with ID starting with 7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9 not found: ID does not exist" containerID="7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"
Oct 31 03:56:34.608292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.600068    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9"} err="failed to get container status \"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9\": rpc error: code = NotFound desc = could not find container \"7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9\": container with ID starting with 7de269d9d72f059d72533658250796df76c373cdbcf2cb40008093b3e1d12ad9 not found: ID does not exist"
Oct 31 03:56:34.608292 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.600092    5831 scope.go:122] "RemoveContainer" containerID="18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"
Oct 31 03:56:34.614124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614088    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614216    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614284 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614252    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614284    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614395    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.614471 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.614454    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.615558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.615253    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.615558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.615321    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\") pod \"e84b7ce9-cced-420b-89f6-b897f79430a0\" (UID: \"e84b7ce9-cced-420b-89f6-b897f79430a0\") "
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.622631    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.622870    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.623058    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.623235    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.625663    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.625987    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.628885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.626363    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.632048 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.630221    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "kube-api-access-v6rql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.632323 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.632296    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo" pod "e84b7ce9-cced-420b-89f6-b897f79430a0" (UID: "e84b7ce9-cced-420b-89f6-b897f79430a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:34.633090 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633064    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633123    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633137    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633151    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7ce9-cced-420b-89f6-b897f79430a0-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633165    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6rql\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-kube-api-access-v6rql\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633180    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0cc-a050-45dc-be8f-e934579c4db0-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633194    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88jsl\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-kube-api-access-88jsl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633207    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633220    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633232    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633246    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633258    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633272    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0cc-a050-45dc-be8f-e934579c4db0-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633285    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7ce9-cced-420b-89f6-b897f79430a0-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633297    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0cc-a050-45dc-be8f-e934579c4db0-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.633641 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.633309    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7ce9-cced-420b-89f6-b897f79430a0-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.639642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.637192    5831 scope.go:122] "RemoveContainer" containerID="18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"
Oct 31 03:56:34.639642 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:34.638319    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d\": container with ID starting with 18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d not found: ID does not exist" containerID="18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"
Oct 31 03:56:34.639642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.638355    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d"} err="failed to get container status \"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d\": rpc error: code = NotFound desc = could not find container \"18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d\": container with ID starting with 18d749c6b083a66a74632c213851faf4c5cf7bb5d46adbb4eb3da6f17d70a04d not found: ID does not exist"
Oct 31 03:56:34.639642 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.638380    5831 scope.go:122] "RemoveContainer" containerID="7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"
Oct 31 03:56:34.652697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.652665    5831 scope.go:122] "RemoveContainer" containerID="7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"
Oct 31 03:56:34.653065 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:34.653043    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f\": container with ID starting with 7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f not found: ID does not exist" containerID="7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"
Oct 31 03:56:34.653174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.653085    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f"} err="failed to get container status \"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f\": rpc error: code = NotFound desc = could not find container \"7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f\": container with ID starting with 7de8d69e610747a21b098ca084d327b96bb991da5ee56625676d89c0264b495f not found: ID does not exist"
Oct 31 03:56:34.653174 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.653104    5831 scope.go:122] "RemoveContainer" containerID="2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"
Oct 31 03:56:34.664157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.664133    5831 scope.go:122] "RemoveContainer" containerID="8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"
Oct 31 03:56:34.674122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.674105    5831 scope.go:122] "RemoveContainer" containerID="2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"
Oct 31 03:56:34.674370 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:34.674353    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92\": container with ID starting with 2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92 not found: ID does not exist" containerID="2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"
Oct 31 03:56:34.674441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.674379    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92"} err="failed to get container status \"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92\": rpc error: code = NotFound desc = could not find container \"2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92\": container with ID starting with 2feec7bce23bc14415c7581e14f8eaf1adb3d89750105cb88ef65ee2fcfced92 not found: ID does not exist"
Oct 31 03:56:34.674441 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.674408    5831 scope.go:122] "RemoveContainer" containerID="8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"
Oct 31 03:56:34.674693 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:34.674671    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf\": container with ID starting with 8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf not found: ID does not exist" containerID="8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"
Oct 31 03:56:34.674762 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.674700    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf"} err="failed to get container status \"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf\": rpc error: code = NotFound desc = could not find container \"8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf\": container with ID starting with 8f77382330b56c0e4e1950199a413bd38d40b60458f593bcff88942c2dfd2caf not found: ID does not exist"
Oct 31 03:56:34.734580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734523    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.734767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734619    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.734767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734656    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.734767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734690    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.734767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734722    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.734905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734780    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.734905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734861    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.734905 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734893    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.735000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734929    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.735000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.734992    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.735068 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735045    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.735130 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735114    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.735181 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735167    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.735215 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735199    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.735334 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735319    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.735393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735379    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.735451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735438    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\") pod \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\" (UID: \"f1dbbd82-7fe2-463a-b5af-b2363d65dcd7\") "
Oct 31 03:56:34.735487 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.735477    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\") pod \"e8bb4852-ad0f-498c-9597-e868ba9556dc\" (UID: \"e8bb4852-ad0f-498c-9597-e868ba9556dc\") "
Oct 31 03:56:34.737600 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.737564    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.737882 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.737853    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.737984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.737869    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.737984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.737958    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.738204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.738121    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.738204 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.738135    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.738649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.738350    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.738649 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.738357    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.739049 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.739033    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.739138 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.739094    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:34.739300 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.739278    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.743126 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.743095    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.743910 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.743862    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "kube-api-access-j4p5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.744112 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.744078    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:34.744398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.744373    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.745005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.744978    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.745110 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.745013    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg" pod "e8bb4852-ad0f-498c-9597-e868ba9556dc" (UID: "e8bb4852-ad0f-498c-9597-e868ba9556dc"). InnerVolumeSpecName "kube-api-access-ck5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.745500 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.745465    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy" pod "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" (UID: "f1dbbd82-7fe2-463a-b5af-b2363d65dcd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.837125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837089    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837120    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb4852-ad0f-498c-9597-e868ba9556dc-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837125 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837130    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837140    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837149    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837158    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck5rg\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-kube-api-access-ck5rg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837167    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837175    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837183    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837191    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837201    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4p5g\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-kube-api-access-j4p5g\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837209    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837217    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837225    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837253    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb4852-ad0f-498c-9597-e868ba9556dc-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837262    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837270    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb4852-ad0f-498c-9597-e868ba9556dc-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.837470 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:34.837279    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:35.075239 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.075209    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:56:35.099547 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.099490    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-68b77"]
Oct 31 03:56:35.177894 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.177802    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:56:35.184344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.184312    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-jxdtb"]
Oct 31 03:56:35.287040 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.287002    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:56:35.293753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.293709    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q"]
Oct 31 03:56:35.396418 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.396373    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:56:35.402064 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.402035    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-gsz9m"]
Oct 31 03:56:35.450247 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.450208    5831 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:56:35.450686 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.450268    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-6j95q" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:56:35.880606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.880512    5831 scope.go:122] "RemoveContainer" containerID="d2bf2b48be01c506d6dbb9815863dc3ec1275489c0115209e7bf1f1ff5d8a34c"
Oct 31 03:56:35.894660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.894635    5831 scope.go:122] "RemoveContainer" containerID="27d1426470b3162e284599ed0d28e64a372981e953eb6b70e5df3ad7ced976e4"
Oct 31 03:56:35.906206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.906183    5831 scope.go:122] "RemoveContainer" containerID="9eeffcb6be384c8e5b004e2f57f8c57317709c921307aa3509e1bbc4f603febc"
Oct 31 03:56:35.918505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.918483    5831 scope.go:122] "RemoveContainer" containerID="43a2fd91420744de9f5174c22f861172004efd79d789e5dc5406ad0cb2ec554f"
Oct 31 03:56:35.928489 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.928468    5831 scope.go:122] "RemoveContainer" containerID="34c53307d528dca1b15c56b7ad56fda7d474408b7ff9695a27a90e675fa4a6f7"
Oct 31 03:56:35.937948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.937931    5831 scope.go:122] "RemoveContainer" containerID="42b9cc97e38f4a94c0e22166897ca3cfd8c1df65c6ca646895d91714bf76ef10"
Oct 31 03:56:35.947243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:35.947222    5831 scope.go:122] "RemoveContainer" containerID="9b818a297aab94651c61961b84b949276de579300c9356ecabde9ca9ca0bff46"
Oct 31 03:56:36.137026 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:36.136939    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" path="/var/lib/kubelet/pods/c686c0cc-a050-45dc-be8f-e934579c4db0/volumes"
Oct 31 03:56:36.137603 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:36.137585    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" path="/var/lib/kubelet/pods/e84b7ce9-cced-420b-89f6-b897f79430a0/volumes"
Oct 31 03:56:36.138051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:36.138038    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" path="/var/lib/kubelet/pods/e8bb4852-ad0f-498c-9597-e868ba9556dc/volumes"
Oct 31 03:56:36.138852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:36.138836    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" path="/var/lib/kubelet/pods/f1dbbd82-7fe2-463a-b5af-b2363d65dcd7/volumes"
Oct 31 03:56:40.536721 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.536685    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 03:56:40.537386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537368    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy"
Oct 31 03:56:40.537386 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537386    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537405    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537410    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537425    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537430    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537455    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537459    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537470    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.537515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537476    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.538013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537623    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.538013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537640    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c686c0cc-a050-45dc-be8f-e934579c4db0" containerName="istio-proxy"
Oct 31 03:56:40.538013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537653    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bb4852-ad0f-498c-9597-e868ba9556dc" containerName="istio-proxy"
Oct 31 03:56:40.538013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537663    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dbbd82-7fe2-463a-b5af-b2363d65dcd7" containerName="istio-proxy"
Oct 31 03:56:40.538013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.537672    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84b7ce9-cced-420b-89f6-b897f79430a0" containerName="istio-proxy"
Oct 31 03:56:40.543656 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.543636    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.593985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.593955    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 03:56:40.622676 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622645    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.622870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622712    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.622870 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622855    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8jh\" (UniqueName: \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.622969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622882    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.622969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622906    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.623063 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.622991    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.635116 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.635049    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"kube-root-ca.crt\""
Oct 31 03:56:40.635602 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.635583    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"default-dockercfg-vk2p6\""
Oct 31 03:56:40.635703 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.635586    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"openshift-service-ca.crt\""
Oct 31 03:56:40.637864 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.637842    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"auth-test-model-502ec2-kserve-self-signed-certs\""
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731341    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8jh\" (UniqueName: \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731397    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731432    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731478    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731665    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.731726    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.732706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.732171    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.733346 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.733275    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.733582 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.733511    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.758327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.750125    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.758327 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.751089    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.785358 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.785324    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8jh\" (UniqueName: \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\") pod \"auth-test-model-502ec2-kserve-69f958db9c-8wckj\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:40.860134 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:40.860039    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:41.236020 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:41.235990    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 03:56:41.238152 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:56:41.238126    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd85d78e_4211_4de9_b36d_93d48725f9bb.slice/crio-0c5e2df7e3808f03e08edae006bf103962440bef0716eda358213a2566df87cc WatchSource:0}: Error finding container 0c5e2df7e3808f03e08edae006bf103962440bef0716eda358213a2566df87cc: Status 404 returned error can't find the container with id 0c5e2df7e3808f03e08edae006bf103962440bef0716eda358213a2566df87cc
Oct 31 03:56:41.652162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:41.652071    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" event={"ID":"fd85d78e-4211-4de9-b36d-93d48725f9bb","Type":"ContainerStarted","Data":"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"}
Oct 31 03:56:41.652162 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:41.652116    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" event={"ID":"fd85d78e-4211-4de9-b36d-93d48725f9bb","Type":"ContainerStarted","Data":"0c5e2df7e3808f03e08edae006bf103962440bef0716eda358213a2566df87cc"}
Oct 31 03:56:41.759446 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:41.759391    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" podStartSLOduration=1.759376966 podStartE2EDuration="1.759376966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:41.756855209 +0000 UTC m=+2951.771416917" watchObservedRunningTime="2026-10-31 03:56:41.759376966 +0000 UTC m=+2951.773938675"
Oct 31 03:56:41.861013 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:41.860981    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:42.980618 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:56:42.980574    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:56:50.860635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:50.860596    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:50.873672 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:50.873643    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:51.753178 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:51.753150    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 03:56:56.066852 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:56:56.066797    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:57:12.979567 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:57:12.979457    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:57:17.498229 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.498191    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:57:17.502719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.502696    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.557141 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.557107    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:57:17.595222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.595191    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dffe64\"/\"kube-root-ca.crt\""
Oct 31 03:57:17.595422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.595304    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dffe64\"/\"openshift-service-ca.crt\""
Oct 31 03:57:17.595422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.595304    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dffe64\"/\"autores-model-dffe64-kserve-self-signed-certs\""
Oct 31 03:57:17.596864 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.596846    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dffe64\"/\"default-dockercfg-dcj5q\""
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668408    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668483    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrzx\" (UniqueName: \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668686    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668743    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668799    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.678558 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.668847    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.769952    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770045 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770027    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770119    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770293 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770161    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrzx\" (UniqueName: \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770319    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770422 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770370    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770572 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770488    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.770759 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770738    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.771005 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.770896    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.772631 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.772603    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.772959 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.772914    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:17.829140 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:17.829109    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrzx\" (UniqueName: \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\") pod \"autores-model-dffe64-kserve-7d8f658477-wnbs7\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") " pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:18.120432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:18.120351    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:18.497512 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:18.497482    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:57:18.499139 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:57:18.499111    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0db17ed4_fec9_4630_a860_0c352b3d1a77.slice/crio-01b615b0b7cb9caaf8de8a961a9194d4c026128edd3d6aad58541ecda200143e WatchSource:0}: Error finding container 01b615b0b7cb9caaf8de8a961a9194d4c026128edd3d6aad58541ecda200143e: Status 404 returned error can't find the container with id 01b615b0b7cb9caaf8de8a961a9194d4c026128edd3d6aad58541ecda200143e
Oct 31 03:57:18.977261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:18.977223    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" event={"ID":"0db17ed4-fec9-4630-a860-0c352b3d1a77","Type":"ContainerStarted","Data":"88eba7a64405977d763d9d8db29b2a8ba054bf846db6acc2185d6f2fbf6d95ec"}
Oct 31 03:57:18.977473 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:18.977272    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" event={"ID":"0db17ed4-fec9-4630-a860-0c352b3d1a77","Type":"ContainerStarted","Data":"01b615b0b7cb9caaf8de8a961a9194d4c026128edd3d6aad58541ecda200143e"}
Oct 31 03:57:19.087000 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:19.086939    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" podStartSLOduration=2.086921755 podStartE2EDuration="2.086921755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:19.082758256 +0000 UTC m=+2989.097319975" watchObservedRunningTime="2026-10-31 03:57:19.086921755 +0000 UTC m=+2989.101483464"
Oct 31 03:57:19.121491 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:19.121449    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:28.121269 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:28.121233    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:28.137170 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:28.137141    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:29.068808 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:29.068776    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:57:34.094937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:34.094898    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:57:36.040565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:36.040511    5831 scope.go:122] "RemoveContainer" containerID="17448b9886d9720af5faba6f41974f928ac07c510564451f81bcef14c606b105"
Oct 31 03:57:38.365832 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.365801    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:57:38.368716 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.368690    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 03:57:38.375562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.375523    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:57:38.378594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.378572    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 03:57:38.379041 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.379022    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:57:38.382252 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:57:38.382236    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 03:57:42.981213 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:57:42.981174    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:58:00.452896 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.452860    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 03:58:00.457753 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.457732    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.510767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.510733    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 03:58:00.550958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.550929    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a5f70c\"/\"kube-root-ca.crt\""
Oct 31 03:58:00.550958 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.550929    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a5f70c\"/\"openshift-service-ca.crt\""
Oct 31 03:58:00.552319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.552297    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a5f70c\"/\"auth-test-model-a5f70c-kserve-self-signed-certs\""
Oct 31 03:58:00.552476 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.552365    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a5f70c\"/\"default-dockercfg-nb6g8\""
Oct 31 03:58:00.571566 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571500    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9d7j\" (UniqueName: \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.571701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571575    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.571701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571675    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.571787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571718    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.571787 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571784    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.571927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.571869    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673131    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9d7j\" (UniqueName: \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673241    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673362 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673284    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673565 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673444    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673635 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673699 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673655    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673769 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673701    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.673865 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.673840    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.675797 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.675770    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.676164 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.676146    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.726806 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.726736    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9d7j\" (UniqueName: \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\") pod \"auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") " pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:00.770963 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:00.770930    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:01.151333 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:01.151308    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 03:58:01.153163 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:58:01.153132    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46d92263_e7ce_4fbf_8f15_8fbe7a3de398.slice/crio-ad874da37459891366522e6418a970b3e456797cfc309da7aceb9e1171dccf8e WatchSource:0}: Error finding container ad874da37459891366522e6418a970b3e456797cfc309da7aceb9e1171dccf8e: Status 404 returned error can't find the container with id ad874da37459891366522e6418a970b3e456797cfc309da7aceb9e1171dccf8e
Oct 31 03:58:01.322767 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:01.322731    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" event={"ID":"46d92263-e7ce-4fbf-8f15-8fbe7a3de398","Type":"ContainerStarted","Data":"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"}
Oct 31 03:58:01.322948 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:01.322782    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" event={"ID":"46d92263-e7ce-4fbf-8f15-8fbe7a3de398","Type":"ContainerStarted","Data":"ad874da37459891366522e6418a970b3e456797cfc309da7aceb9e1171dccf8e"}
Oct 31 03:58:01.431206 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:01.431155    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" podStartSLOduration=1.431140651 podStartE2EDuration="1.431140651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:01.427958536 +0000 UTC m=+3031.442520243" watchObservedRunningTime="2026-10-31 03:58:01.431140651 +0000 UTC m=+3031.445702380"
Oct 31 03:58:01.775074 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:01.774963    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:03.061357 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:03.061298    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:58:03.061941 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:03.061523    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" containerID="cri-o://556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42" gracePeriod=30
Oct 31 03:58:05.320417 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.320370    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 03:58:05.325227 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.325207    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.376647 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.376612    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 03:58:05.417594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.417514    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ab9b62\"/\"openshift-service-ca.crt\""
Oct 31 03:58:05.417594 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.417514    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ab9b62\"/\"kube-root-ca.crt\""
Oct 31 03:58:05.417887 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.417646    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ab9b62\"/\"default-dockercfg-vmbbd\""
Oct 31 03:58:05.419221 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.419200    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ab9b62\"/\"praxis-route-ab9b62-kserve-self-signed-certs\""
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468176    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468232    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468311    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468502    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lxm\" (UniqueName: \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468583    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.479574 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.468637    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.569969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.569926    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570084    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lxm\" (UniqueName: \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570124    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570157    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570190    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570298 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570210    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570599    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570620    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.570777 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.570670    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.572571 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.572517    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.572719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.572656    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.623701 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.623672    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lxm\" (UniqueName: \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\") pod \"praxis-route-ab9b62-kserve-546dc998d5-w5lwc\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") " pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:05.639581 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:05.639538    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:06.021554 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:06.021463    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 03:58:06.026871 ip-10-0-140-194 kubenswrapper[5831]: W1031 03:58:06.026840    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52550c21_6da5_4a9b_81ac_e00b9e8d1805.slice/crio-54322ee8fa3f2ca8ae9bce20da5784fa2f84fb0b6e7c67a9b23de96829e44035 WatchSource:0}: Error finding container 54322ee8fa3f2ca8ae9bce20da5784fa2f84fb0b6e7c67a9b23de96829e44035: Status 404 returned error can't find the container with id 54322ee8fa3f2ca8ae9bce20da5784fa2f84fb0b6e7c67a9b23de96829e44035
Oct 31 03:58:06.364544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:06.364443    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" event={"ID":"52550c21-6da5-4a9b-81ac-e00b9e8d1805","Type":"ContainerStarted","Data":"6ed597fdb0dcb51ca5a7befd19ba430f636699d35ea3548cf44a79227baffaac"}
Oct 31 03:58:06.364544 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:06.364489    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" event={"ID":"52550c21-6da5-4a9b-81ac-e00b9e8d1805","Type":"ContainerStarted","Data":"54322ee8fa3f2ca8ae9bce20da5784fa2f84fb0b6e7c67a9b23de96829e44035"}
Oct 31 03:58:06.473792 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:06.473729    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" podStartSLOduration=1.473714167 podStartE2EDuration="1.473714167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:06.470179287 +0000 UTC m=+3036.484740994" watchObservedRunningTime="2026-10-31 03:58:06.473714167 +0000 UTC m=+3036.488275875"
Oct 31 03:58:06.641505 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:06.641412    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:07.756778 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:07.756740    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:58:07.757260 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:07.756801    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:58:08.366951 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.366924    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:58:08.381690 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.381654    5831 generic.go:356] "Generic (PLEG): container finished" podID="613fdee2-7f61-4328-af15-65a992d2351a" containerID="556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42" exitCode=0
Oct 31 03:58:08.381967 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.381714    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"
Oct 31 03:58:08.382122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.381723    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" event={"ID":"613fdee2-7f61-4328-af15-65a992d2351a","Type":"ContainerDied","Data":"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"}
Oct 31 03:58:08.382122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.382073    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7" event={"ID":"613fdee2-7f61-4328-af15-65a992d2351a","Type":"ContainerDied","Data":"27fc79474fa3641d9203e72bc152a0ab2ed627ca007ec78787e765e3008574ef"}
Oct 31 03:58:08.382122 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.382111    5831 scope.go:122] "RemoveContainer" containerID="556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"
Oct 31 03:58:08.394673 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.394648    5831 scope.go:122] "RemoveContainer" containerID="556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"
Oct 31 03:58:08.395007 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:58:08.394985    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42\": container with ID starting with 556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42 not found: ID does not exist" containerID="556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"
Oct 31 03:58:08.395075 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.395020    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42"} err="failed to get container status \"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42\": rpc error: code = NotFound desc = could not find container \"556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42\": container with ID starting with 556af7d83708b59bb7ce441abe70fcf915473a335b34e34813d47ab2de958a42 not found: ID does not exist"
Oct 31 03:58:08.456180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456139    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456205    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456393 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456295    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456394    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456491    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:08.456704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456572    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456626    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.456704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456704 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456649    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.456937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456714    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456748    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456793    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\") pod \"613fdee2-7f61-4328-af15-65a992d2351a\" (UID: \"613fdee2-7f61-4328-af15-65a992d2351a\") "
Oct 31 03:58:08.456937 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.456887    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.457145 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457092    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.457374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457351    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.457374 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457374    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.457585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457387    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613fdee2-7f61-4328-af15-65a992d2351a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.457585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457401    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.457585 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.457413    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.458756 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.458729    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:08.459261 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.459241    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:08.459403 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.459391    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:08.459469 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.459394    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4" pod "613fdee2-7f61-4328-af15-65a992d2351a" (UID: "613fdee2-7f61-4328-af15-65a992d2351a"). InnerVolumeSpecName "kube-api-access-lk9j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:08.558846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.558810    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9j4\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-kube-api-access-lk9j4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.558846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.558841    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613fdee2-7f61-4328-af15-65a992d2351a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.558846 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.558851    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613fdee2-7f61-4328-af15-65a992d2351a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.559180 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.558861    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613fdee2-7f61-4328-af15-65a992d2351a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:08.800127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.800076    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:58:08.805794 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:08.805766    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-22nj7"]
Oct 31 03:58:09.046816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:09.046730    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"]
Oct 31 03:58:09.050453 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:09.050425    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3944492-dpn7b"]
Oct 31 03:58:10.137154 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:10.137121    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d283016-5cd7-4602-88fb-31f156e52a29" path="/var/lib/kubelet/pods/0d283016-5cd7-4602-88fb-31f156e52a29/volumes"
Oct 31 03:58:10.137684 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:10.137668    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613fdee2-7f61-4328-af15-65a992d2351a" path="/var/lib/kubelet/pods/613fdee2-7f61-4328-af15-65a992d2351a/volumes"
Oct 31 03:58:10.771927 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:10.771897    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:10.784660 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:10.784632    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:11.422630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:11.422595    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 03:58:12.979364 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:58:12.979322    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:58:14.985072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:14.985031    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:58:15.639830 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:15.639791    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:15.652854 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:15.652828    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:16.474784 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:16.474757    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 03:58:20.046961 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:20.046928    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"]
Oct 31 03:58:20.093801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:20.093769    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c2b868-88nb6"]
Oct 31 03:58:20.137462 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:20.137428    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953bf90c-14d4-4c33-8d48-dfc8bf3351c6" path="/var/lib/kubelet/pods/953bf90c-14d4-4c33-8d48-dfc8bf3351c6/volumes"
Oct 31 03:58:42.979359 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:58:42.979324    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:58:48.588176 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:58:48.588142    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:59:03.520354 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.520315    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:59:03.520985 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.520562    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" containerID="cri-o://d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29" gracePeriod=30
Oct 31 03:59:03.612570 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.610159    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" podUID="0db17ed4-fec9-4630-a860-0c352b3d1a77" containerName="main" containerID="cri-o://88eba7a64405977d763d9d8db29b2a8ba054bf846db6acc2185d6f2fbf6d95ec" gracePeriod=45
Oct 31 03:59:03.860461 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.860426    5831 generic.go:356] "Generic (PLEG): container finished" podID="0db17ed4-fec9-4630-a860-0c352b3d1a77" containerID="88eba7a64405977d763d9d8db29b2a8ba054bf846db6acc2185d6f2fbf6d95ec" exitCode=0
Oct 31 03:59:03.860658 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.860493    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" event={"ID":"0db17ed4-fec9-4630-a860-0c352b3d1a77","Type":"ContainerDied","Data":"88eba7a64405977d763d9d8db29b2a8ba054bf846db6acc2185d6f2fbf6d95ec"}
Oct 31 03:59:03.917051 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:03.917021    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:59:04.020251 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020215    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.020451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020329    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\" (UniqueName: \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.020451 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020427    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.020623 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\" (UniqueName: \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.020719 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020701    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.020835 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020809    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:04.021003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020952    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:04.021003 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.020990    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\") pod \"0db17ed4-fec9-4630-a860-0c352b3d1a77\" (UID: \"0db17ed4-fec9-4630-a860-0c352b3d1a77\") "
Oct 31 03:59:04.021330 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.021076    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:04.021630 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.021609    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.021726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.021634    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.021726 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.021647    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.023151 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.023125    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "kube-api-access-hkrzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:04.023262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.023218    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:04.023262 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.023242    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs" pod "0db17ed4-fec9-4630-a860-0c352b3d1a77" (UID: "0db17ed4-fec9-4630-a860-0c352b3d1a77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:04.123337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.123238    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrzx\" (UniqueName: \"kubernetes.io/projected/0db17ed4-fec9-4630-a860-0c352b3d1a77-kube-api-access-hkrzx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.123337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.123281    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db17ed4-fec9-4630-a860-0c352b3d1a77-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.123337 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.123296    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0db17ed4-fec9-4630-a860-0c352b3d1a77-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.872344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.872316    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"
Oct 31 03:59:04.872344 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.872321    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7" event={"ID":"0db17ed4-fec9-4630-a860-0c352b3d1a77","Type":"ContainerDied","Data":"01b615b0b7cb9caaf8de8a961a9194d4c026128edd3d6aad58541ecda200143e"}
Oct 31 03:59:04.872969 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:04.872391    5831 scope.go:122] "RemoveContainer" containerID="88eba7a64405977d763d9d8db29b2a8ba054bf846db6acc2185d6f2fbf6d95ec"
Oct 31 03:59:05.028297 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:05.028260    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:59:05.073890 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:05.073842    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-dffe64/autores-model-dffe64-kserve-7d8f658477-wnbs7"]
Oct 31 03:59:06.138072 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:06.138036    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db17ed4-fec9-4630-a860-0c352b3d1a77" path="/var/lib/kubelet/pods/0db17ed4-fec9-4630-a860-0c352b3d1a77/volumes"
Oct 31 03:59:08.391094 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.391052    5831 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:59:08.391706 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.391134    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:59:08.489035 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.488994    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 03:59:08.826984 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.826957    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:59:08.892606 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.892816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892627    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.892816 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892715    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.892964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892817    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.892964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892897    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.892964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892958    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.893179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.892986    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.893179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.893072    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.893179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.893121    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.893380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.893233    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\") pod \"7be57f7a-f741-46a7-8101-3e7aac68daee\" (UID: \"7be57f7a-f741-46a7-8101-3e7aac68daee\") "
Oct 31 03:59:08.893380 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.893258    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.893562 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.893507    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.894083 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.894055    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.894088    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894222 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.894105    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894352 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.894301    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:08.894651 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.894625    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.895466 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.895436    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:08.895674 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.895649    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "kube-api-access-jfnct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:08.895761 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.895670    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.896179 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.896161    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token" pod "7be57f7a-f741-46a7-8101-3e7aac68daee" (UID: "7be57f7a-f741-46a7-8101-3e7aac68daee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:08.912226 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.912195    5831 generic.go:356] "Generic (PLEG): container finished" podID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerID="d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29" exitCode=0
Oct 31 03:59:08.912348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.912226    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" event={"ID":"7be57f7a-f741-46a7-8101-3e7aac68daee","Type":"ContainerDied","Data":"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"}
Oct 31 03:59:08.912348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.912260    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"
Oct 31 03:59:08.912348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.912284    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn" event={"ID":"7be57f7a-f741-46a7-8101-3e7aac68daee","Type":"ContainerDied","Data":"1da569f46fef12ccfe6e7e146d777dc3a9db55e99f2abfc928ac2efcd861b6ad"}
Oct 31 03:59:08.912348 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.912311    5831 scope.go:122] "RemoveContainer" containerID="d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"
Oct 31 03:59:08.936697 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.936667    5831 scope.go:122] "RemoveContainer" containerID="d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"
Oct 31 03:59:08.937075 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:59:08.937054    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29\": container with ID starting with d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29 not found: ID does not exist" containerID="d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"
Oct 31 03:59:08.937157 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.937084    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29"} err="failed to get container status \"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29\": rpc error: code = NotFound desc = could not find container \"d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29\": container with ID starting with d6c67db655c9d6a2bb9eea10cea5acdc1311421f3ad2750d5939d3215f81ad29 not found: ID does not exist"
Oct 31 03:59:08.995580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995503    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995580 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995584    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995598    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnct\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-kube-api-access-jfnct\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995608    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be57f7a-f741-46a7-8101-3e7aac68daee-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995616    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be57f7a-f741-46a7-8101-3e7aac68daee-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995801 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:08.995626    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be57f7a-f741-46a7-8101-3e7aac68daee-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:09.066869 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:09.066834    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:59:09.070971 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:09.070930    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-nd7dn"]
Oct 31 03:59:10.138515 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:10.138475    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" path="/var/lib/kubelet/pods/7be57f7a-f741-46a7-8101-3e7aac68daee/volumes"
Oct 31 03:59:12.979909 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:59:12.979868    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:59:31.573785 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:31.573738    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:59:36.136713 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.136680    5831 scope.go:122] "RemoveContainer" containerID="508b2f3b2b343686ca21c115102702838aa866837d8ae0f5785666fd90c96232"
Oct 31 03:59:36.147243 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.147219    5831 scope.go:122] "RemoveContainer" containerID="c5e971b8ed5889e0e52375974280063ee60e22beb75a8d62ec03a147127a7c1a"
Oct 31 03:59:36.574556 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.574493    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" podUID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" containerName="payload-processing" containerID="cri-o://f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277" gracePeriod=25
Oct 31 03:59:36.849217 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.849191    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:59:36.918127 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.918096    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\") pod \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") "
Oct 31 03:59:36.918324 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.918229    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\" (UniqueName: \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\") pod \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\" (UID: \"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361\") "
Oct 31 03:59:36.918521 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.918497    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume" pod "9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" (UID: "9bedd4ef-617f-4a03-b0f0-51ee6a6c2361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:36.918812 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.918792    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:36.920507 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:36.920482    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq" pod "9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" (UID: "9bedd4ef-617f-4a03-b0f0-51ee6a6c2361"). InnerVolumeSpecName "kube-api-access-qmfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:37.019875 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.019843    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfgq\" (UniqueName: \"kubernetes.io/projected/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361-kube-api-access-qmfgq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:37.232018 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.231983    5831 generic.go:356] "Generic (PLEG): container finished" podID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" containerID="f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277" exitCode=0
Oct 31 03:59:37.232614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.232040    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"
Oct 31 03:59:37.232614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.232071    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" event={"ID":"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361","Type":"ContainerDied","Data":"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"}
Oct 31 03:59:37.232614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.232154    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg" event={"ID":"9bedd4ef-617f-4a03-b0f0-51ee6a6c2361","Type":"ContainerDied","Data":"ee379d7556046f613a3d929e509ae4968f0d650f7eeb22a6fa47ec39281edbae"}
Oct 31 03:59:37.232614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.232199    5831 scope.go:122] "RemoveContainer" containerID="f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"
Oct 31 03:59:37.246614 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.246598    5831 scope.go:122] "RemoveContainer" containerID="f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"
Oct 31 03:59:37.246892 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:59:37.246872    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277\": container with ID starting with f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277 not found: ID does not exist" containerID="f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"
Oct 31 03:59:37.246964 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.246899    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277"} err="failed to get container status \"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277\": rpc error: code = NotFound desc = could not find container \"f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277\": container with ID starting with f6c01509c513ea4609ce9d282b8054f9d3bfb3ad60529dbbf25428b8113ab277 not found: ID does not exist"
Oct 31 03:59:37.387919 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.387885    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:59:37.433824 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:37.433797    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dffe64-6b794cbd59-ldmrg"]
Oct 31 03:59:38.138463 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:38.138426    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" path="/var/lib/kubelet/pods/9bedd4ef-617f-4a03-b0f0-51ee6a6c2361/volumes"
Oct 31 03:59:42.979405 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:59:42.979360    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 03:59:53.179850 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:53.179815    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 03:59:53.180282 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:53.180032    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy" containerID="cri-o://5730ca211f6a81bda97794ec7272a628794be01d1e66948608394fc739c69801" gracePeriod=30
Oct 31 03:59:58.439398 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.439365    5831 generic.go:356] "Generic (PLEG): container finished" podID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerID="5730ca211f6a81bda97794ec7272a628794be01d1e66948608394fc739c69801" exitCode=0
Oct 31 03:59:58.439912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.439444    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerDied","Data":"5730ca211f6a81bda97794ec7272a628794be01d1e66948608394fc739c69801"}
Oct 31 03:59:58.439912 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.439506    5831 scope.go:122] "RemoveContainer" containerID="0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03"
Oct 31 03:59:58.472439 ip-10-0-140-194 kubenswrapper[5831]: E1031 03:59:58.472393    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03\": container with ID starting with 0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03 not found: ID does not exist" containerID="0104411c1fedb9a6068852a2d06557ac2caf287b7c63589e0e06c663ca063c03"
Oct 31 03:59:58.476167 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.476144    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:59:58.522189 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522155    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522204    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522251    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522320    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522357    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522387 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522374    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522394    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522433    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522462    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\") pod \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\" (UID: \"b38e149c-aca7-43d5-a10d-b5a41216b9ba\") "
Oct 31 03:59:58.522709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522466    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.522709 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522542    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.522999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522716    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.522999 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.522922    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.523121 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523078    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:58.523232 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523212    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.523304 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523244    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.523432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523365    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.523432 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523430    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.523687 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.523444    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.525033 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.525007    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:58.525135 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.525111    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.525248 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.525231    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "kube-api-access-f2dnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:58.525307 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.525269    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo" pod "b38e149c-aca7-43d5-a10d-b5a41216b9ba" (UID: "b38e149c-aca7-43d5-a10d-b5a41216b9ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:58.624885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.624850    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.624885 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.624882    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.625124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.624984    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dnq\" (UniqueName: \"kubernetes.io/projected/b38e149c-aca7-43d5-a10d-b5a41216b9ba-kube-api-access-f2dnq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.625124 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:58.625002    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e149c-aca7-43d5-a10d-b5a41216b9ba-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:59.449172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:59.449141    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"
Oct 31 03:59:59.449172 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:59.449155    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt" event={"ID":"b38e149c-aca7-43d5-a10d-b5a41216b9ba","Type":"ContainerDied","Data":"cfc91a7d81d62eabf4571506c6f2e986cb387edea05d641ac46e2b8df651152c"}
Oct 31 03:59:59.449773 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:59.449209    5831 scope.go:122] "RemoveContainer" containerID="5730ca211f6a81bda97794ec7272a628794be01d1e66948608394fc739c69801"
Oct 31 03:59:59.649819 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:59.649783    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 03:59:59.695319 ip-10-0-140-194 kubenswrapper[5831]: I1031 03:59:59.695280    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-dffe64-openshift-default-58c6cb4585-d2xzt"]
Oct 31 04:00:00.136120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.136087    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" path="/var/lib/kubelet/pods/b38e149c-aca7-43d5-a10d-b5a41216b9ba/volumes"
Oct 31 04:00:00.204843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.204810    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:00.205434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205421    5831 cpu_manager.go:414] "containerMap: removing container" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy"
Oct 31 04:00:00.205484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205436    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy"
Oct 31 04:00:00.205484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205463    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy"
Oct 31 04:00:00.205484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205468    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy"
Oct 31 04:00:00.205484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205475    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.205484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205480    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205492    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0db17ed4-fec9-4630-a860-0c352b3d1a77" containerName="main"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205497    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db17ed4-fec9-4630-a860-0c352b3d1a77" containerName="main"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205504    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205509    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205521    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" containerName="payload-processing"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205540    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" containerName="payload-processing"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205677    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bedd4ef-617f-4a03-b0f0-51ee6a6c2361" containerName="payload-processing"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205690    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="613fdee2-7f61-4328-af15-65a992d2351a" containerName="istio-proxy"
Oct 31 04:00:00.205720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205720    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be57f7a-f741-46a7-8101-3e7aac68daee" containerName="istio-proxy"
Oct 31 04:00:00.206151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205727    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.206151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205734    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db17ed4-fec9-4630-a860-0c352b3d1a77" containerName="main"
Oct 31 04:00:00.206151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.205739    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38e149c-aca7-43d5-a10d-b5a41216b9ba" containerName="istio-proxy"
Oct 31 04:00:00.210924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.210908    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.216615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.216578    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:00.221321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.221303    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.250256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.250230    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:00.254584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.254564    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:00.254693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.254612    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.258658 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.258640    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.264465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.264014    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:00:00.283123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.283094    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:00.298327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.291820    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.300166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.299880    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:00.306567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.306541    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:00.344902 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.344834    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:00.346492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.346459    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwjt\" (UniqueName: \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.346786 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.346757    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.346908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.346800    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mcg\" (UniqueName: \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.346908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.346839    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfw5h\" (UniqueName: \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.346998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.346983    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.347038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.347017    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.347075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.347045    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjc88\" (UniqueName: \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.347143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.347129    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.349578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.349555    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:00.360436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.360402    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:00.448523 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448491    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.448737 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448584    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.448737 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448618    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.448737 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448640    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjc88\" (UniqueName: \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.448930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448765    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.448930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448863    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwjt\" (UniqueName: \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.449050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448927    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bjl\" (UniqueName: \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.449050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448961    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.449050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.448990    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mcg\" (UniqueName: \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.449220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.449053    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfw5h\" (UniqueName: \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.449652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.449389    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.449652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.449591    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.449652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.449605    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.449828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.449718    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.501893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.501854    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwjt\" (UniqueName: \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.502060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.501906    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfw5h\" (UniqueName: \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.502144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.502123    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjc88\" (UniqueName: \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.502433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.502411    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mcg\" (UniqueName: \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.526703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.526679    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:00.535182 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.535154    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:00.551093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.551056    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bjl\" (UniqueName: \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.551572 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.551250    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.552073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.552053    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.571812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.571783    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:00.583610 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.583580    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:00.606963 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.606937    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bjl\" (UniqueName: \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.612682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.611771    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:00.918030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.917997    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:00.920632 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:00:00.920588    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a82ece1_0234_4fe0_a827_a1fa6881319b.slice/crio-a4084387d7014ef9fcef3adf611c37a163ce0b705563e3b6c84dc81110bf8f7f WatchSource:0}: Error finding container a4084387d7014ef9fcef3adf611c37a163ce0b705563e3b6c84dc81110bf8f7f: Status 404 returned error can't find the container with id a4084387d7014ef9fcef3adf611c37a163ce0b705563e3b6c84dc81110bf8f7f
Oct 31 04:00:00.966703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:00.966673    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:00.967383 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:00:00.967358    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89ad831_bed1_474d_a852_c8230aba7eac.slice/crio-7481a48c85f65d5880bc57ed0563118e3bd80729ee8470dbd62bd24cc9258394 WatchSource:0}: Error finding container 7481a48c85f65d5880bc57ed0563118e3bd80729ee8470dbd62bd24cc9258394: Status 404 returned error can't find the container with id 7481a48c85f65d5880bc57ed0563118e3bd80729ee8470dbd62bd24cc9258394
Oct 31 04:00:01.016475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.016444    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:01.022109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.022081    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:01.028473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.028447    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:01.082486 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:00:01.082452    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a59bad_9153_4c9a_bb71_17e3f0e20567.slice/crio-740c69a21c766f53f59e2de6761fdafa4433703942b81d6621d681ff568077cc WatchSource:0}: Error finding container 740c69a21c766f53f59e2de6761fdafa4433703942b81d6621d681ff568077cc: Status 404 returned error can't find the container with id 740c69a21c766f53f59e2de6761fdafa4433703942b81d6621d681ff568077cc
Oct 31 04:00:01.466546 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.466497    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerStarted","Data":"740c69a21c766f53f59e2de6761fdafa4433703942b81d6621d681ff568077cc"}
Oct 31 04:00:01.468139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.468112    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerStarted","Data":"515f8e0824c22801d8ce97b91e837e32888776b089163679e471ec22fca53da8"}
Oct 31 04:00:01.469928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.469894    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerStarted","Data":"035bfc8912cad4f687fc7a13250755b7835559fec690bc1aad8806207149e6df"}
Oct 31 04:00:01.470041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.469942    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerStarted","Data":"a4084387d7014ef9fcef3adf611c37a163ce0b705563e3b6c84dc81110bf8f7f"}
Oct 31 04:00:01.471280 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.471245    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerStarted","Data":"55063738592ffe883f2b6d100840741d6ed73cdb7abbcf6994aa8d8f2ef71bbe"}
Oct 31 04:00:01.472777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.472756    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerStarted","Data":"7481a48c85f65d5880bc57ed0563118e3bd80729ee8470dbd62bd24cc9258394"}
Oct 31 04:00:01.576195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:01.576073    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" podStartSLOduration=1.5760579799999999 podStartE2EDuration="1.57605798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.574516506 +0000 UTC m=+3151.589078217" watchObservedRunningTime="2026-10-31 04:00:01.57605798 +0000 UTC m=+3151.590619687"
Oct 31 04:00:02.484251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.484215    5831 generic.go:356] "Generic (PLEG): container finished" podID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerID="9d03c30e46760b6936ac2242a1311f841430a88e2dd05869acd5a1f8ea049ba2" exitCode=60
Oct 31 04:00:02.484868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.484309    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerDied","Data":"9d03c30e46760b6936ac2242a1311f841430a88e2dd05869acd5a1f8ea049ba2"}
Oct 31 04:00:02.484868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.484731    5831 scope.go:122] "RemoveContainer" containerID="9d03c30e46760b6936ac2242a1311f841430a88e2dd05869acd5a1f8ea049ba2"
Oct 31 04:00:02.486367 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.486340    5831 generic.go:356] "Generic (PLEG): container finished" podID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerID="035bfc8912cad4f687fc7a13250755b7835559fec690bc1aad8806207149e6df" exitCode=60
Oct 31 04:00:02.486564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.486426    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerDied","Data":"035bfc8912cad4f687fc7a13250755b7835559fec690bc1aad8806207149e6df"}
Oct 31 04:00:02.486728 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.486710    5831 scope.go:122] "RemoveContainer" containerID="035bfc8912cad4f687fc7a13250755b7835559fec690bc1aad8806207149e6df"
Oct 31 04:00:02.488305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.488263    5831 generic.go:356] "Generic (PLEG): container finished" podID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerID="28219ce9866612d1276fef0467ce06d2083540b05a5f0a5bb110eeb367d0af48" exitCode=60
Oct 31 04:00:02.488381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.488334    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerDied","Data":"28219ce9866612d1276fef0467ce06d2083540b05a5f0a5bb110eeb367d0af48"}
Oct 31 04:00:02.488588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.488574    5831 scope.go:122] "RemoveContainer" containerID="28219ce9866612d1276fef0467ce06d2083540b05a5f0a5bb110eeb367d0af48"
Oct 31 04:00:02.491410 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.491388    5831 generic.go:356] "Generic (PLEG): container finished" podID="e89ad831-bed1-474d-a852-c8230aba7eac" containerID="e7fc114f42e811c87d9cdbb9b08ba2a2f617c870a09cdd1702acdf23481d2ee4" exitCode=60
Oct 31 04:00:02.491500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.491473    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerDied","Data":"e7fc114f42e811c87d9cdbb9b08ba2a2f617c870a09cdd1702acdf23481d2ee4"}
Oct 31 04:00:02.491651 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.491637    5831 scope.go:122] "RemoveContainer" containerID="e7fc114f42e811c87d9cdbb9b08ba2a2f617c870a09cdd1702acdf23481d2ee4"
Oct 31 04:00:02.493108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.493088    5831 generic.go:356] "Generic (PLEG): container finished" podID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerID="c83d0760bde755b936d10cae08aef6dae427a03e292711e518c67d8780fa475c" exitCode=60
Oct 31 04:00:02.493230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.493177    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerDied","Data":"c83d0760bde755b936d10cae08aef6dae427a03e292711e518c67d8780fa475c"}
Oct 31 04:00:02.493366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:02.493350    5831 scope.go:122] "RemoveContainer" containerID="c83d0760bde755b936d10cae08aef6dae427a03e292711e518c67d8780fa475c"
Oct 31 04:00:03.504744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.504710    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerStarted","Data":"d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"}
Oct 31 04:00:03.506836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.506802    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerStarted","Data":"45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"}
Oct 31 04:00:03.508834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.508805    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerStarted","Data":"7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"}
Oct 31 04:00:03.511061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.511036    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerStarted","Data":"35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"}
Oct 31 04:00:03.512985 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.512956    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerStarted","Data":"24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"}
Oct 31 04:00:03.610347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.610290    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" podStartSLOduration=3.610276303 podStartE2EDuration="3.610276303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.609760142 +0000 UTC m=+3153.624321853" watchObservedRunningTime="2026-10-31 04:00:03.610276303 +0000 UTC m=+3153.624838011"
Oct 31 04:00:03.722070 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.722009    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" podStartSLOduration=3.721994029 podStartE2EDuration="3.721994029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.720851168 +0000 UTC m=+3153.735412887" watchObservedRunningTime="2026-10-31 04:00:03.721994029 +0000 UTC m=+3153.736555736"
Oct 31 04:00:03.828999 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:03.828935    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" podStartSLOduration=3.82892 podStartE2EDuration="3.82892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.825990367 +0000 UTC m=+3153.840552075" watchObservedRunningTime="2026-10-31 04:00:03.82892 +0000 UTC m=+3153.843481708"
Oct 31 04:00:04.056148 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.056040    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" podStartSLOduration=4.056025062 podStartE2EDuration="4.056025062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:04.054375255 +0000 UTC m=+3154.068936962" watchObservedRunningTime="2026-10-31 04:00:04.056025062 +0000 UTC m=+3154.070586771"
Oct 31 04:00:04.523197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.523160    5831 generic.go:356] "Generic (PLEG): container finished" podID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerID="35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad" exitCode=60
Oct 31 04:00:04.523740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.523247    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerDied","Data":"35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"}
Oct 31 04:00:04.523740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.523311    5831 scope.go:122] "RemoveContainer" containerID="9d03c30e46760b6936ac2242a1311f841430a88e2dd05869acd5a1f8ea049ba2"
Oct 31 04:00:04.523740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.523694    5831 scope.go:122] "RemoveContainer" containerID="35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"
Oct 31 04:00:04.524081 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:04.524058    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj_odh-ai-gateway-infra(0245e787-0d08-4c8b-8af5-86ef5b363e6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f"
Oct 31 04:00:04.525991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.525969    5831 generic.go:356] "Generic (PLEG): container finished" podID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerID="24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6" exitCode=60
Oct 31 04:00:04.526110 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.526046    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerDied","Data":"24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"}
Oct 31 04:00:04.526359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.526341    5831 scope.go:122] "RemoveContainer" containerID="24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"
Oct 31 04:00:04.526720 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:04.526696    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq_odh-ai-gateway-infra(8a82ece1-0234-4fe0-a827-a1fa6881319b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b"
Oct 31 04:00:04.528481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.528376    5831 generic.go:356] "Generic (PLEG): container finished" podID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerID="d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa" exitCode=60
Oct 31 04:00:04.528481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.528448    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerDied","Data":"d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"}
Oct 31 04:00:04.528811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.528792    5831 scope.go:122] "RemoveContainer" containerID="d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"
Oct 31 04:00:04.529090 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:04.529071    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s_odh-ai-gateway-infra(57f77ad0-4af7-46eb-a782-76110ea378bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf"
Oct 31 04:00:04.530788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.530769    5831 generic.go:356] "Generic (PLEG): container finished" podID="e89ad831-bed1-474d-a852-c8230aba7eac" containerID="45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c" exitCode=60
Oct 31 04:00:04.530909 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.530862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerDied","Data":"45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"}
Oct 31 04:00:04.531107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.531091    5831 scope.go:122] "RemoveContainer" containerID="45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"
Oct 31 04:00:04.531411 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:04.531387    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd_odh-ai-gateway-infra(e89ad831-bed1-474d-a852-c8230aba7eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" podUID="e89ad831-bed1-474d-a852-c8230aba7eac"
Oct 31 04:00:04.533024 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.533002    5831 generic.go:356] "Generic (PLEG): container finished" podID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerID="7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e" exitCode=60
Oct 31 04:00:04.533109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.533079    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerDied","Data":"7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"}
Oct 31 04:00:04.533287 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.533272    5831 scope.go:122] "RemoveContainer" containerID="7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"
Oct 31 04:00:04.533516 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:04.533499    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774_odh-ai-gateway-infra(51a59bad-9153-4c9a-bb71-17e3f0e20567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567"
Oct 31 04:00:04.539256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.539238    5831 scope.go:122] "RemoveContainer" containerID="035bfc8912cad4f687fc7a13250755b7835559fec690bc1aad8806207149e6df"
Oct 31 04:00:04.553923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.553902    5831 scope.go:122] "RemoveContainer" containerID="28219ce9866612d1276fef0467ce06d2083540b05a5f0a5bb110eeb367d0af48"
Oct 31 04:00:04.568988 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.568960    5831 scope.go:122] "RemoveContainer" containerID="e7fc114f42e811c87d9cdbb9b08ba2a2f617c870a09cdd1702acdf23481d2ee4"
Oct 31 04:00:04.588478 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:04.588448    5831 scope.go:122] "RemoveContainer" containerID="c83d0760bde755b936d10cae08aef6dae427a03e292711e518c67d8780fa475c"
Oct 31 04:00:05.544221 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:05.544189    5831 scope.go:122] "RemoveContainer" containerID="35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"
Oct 31 04:00:05.544730 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:05.544595    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj_odh-ai-gateway-infra(0245e787-0d08-4c8b-8af5-86ef5b363e6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f"
Oct 31 04:00:05.546060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:05.546039    5831 scope.go:122] "RemoveContainer" containerID="24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"
Oct 31 04:00:05.546355 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:05.546334    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq_odh-ai-gateway-infra(8a82ece1-0234-4fe0-a827-a1fa6881319b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b"
Oct 31 04:00:05.547790 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:05.547766    5831 scope.go:122] "RemoveContainer" containerID="d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"
Oct 31 04:00:05.548069 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:05.548040    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s_odh-ai-gateway-infra(57f77ad0-4af7-46eb-a782-76110ea378bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf"
Oct 31 04:00:05.549415 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:05.549399    5831 scope.go:122] "RemoveContainer" containerID="45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"
Oct 31 04:00:05.549675 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:05.549655    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd_odh-ai-gateway-infra(e89ad831-bed1-474d-a852-c8230aba7eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" podUID="e89ad831-bed1-474d-a852-c8230aba7eac"
Oct 31 04:00:05.551024 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:05.551007    5831 scope.go:122] "RemoveContainer" containerID="7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"
Oct 31 04:00:05.551217 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:05.551200    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774_odh-ai-gateway-infra(51a59bad-9153-4c9a-bb71-17e3f0e20567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567"
Oct 31 04:00:12.979041 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:12.978996    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:00:14.030254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:14.030216    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:00:15.128073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:15.128043    5831 scope.go:122] "RemoveContainer" containerID="35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"
Oct 31 04:00:15.128723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:15.128191    5831 scope.go:122] "RemoveContainer" containerID="24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"
Oct 31 04:00:15.633910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:15.633876    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerStarted","Data":"294b73fc015fa1c24de3e50be44d733e0d8807329971f376526ac98d44a1c623"}
Oct 31 04:00:15.636058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:15.636031    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerStarted","Data":"e9d31fbae0b2b7841eeacfb202a06ac3a862cf17d4b0c083ffa0a99ef87153c6"}
Oct 31 04:00:16.128199 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.128161    5831 scope.go:122] "RemoveContainer" containerID="7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"
Oct 31 04:00:16.645361 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.645324    5831 generic.go:356] "Generic (PLEG): container finished" podID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerID="e9d31fbae0b2b7841eeacfb202a06ac3a862cf17d4b0c083ffa0a99ef87153c6" exitCode=60
Oct 31 04:00:16.645603 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.645396    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerDied","Data":"e9d31fbae0b2b7841eeacfb202a06ac3a862cf17d4b0c083ffa0a99ef87153c6"}
Oct 31 04:00:16.645603 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.645458    5831 scope.go:122] "RemoveContainer" containerID="35ade12cfa1814f15758fcde27ae850e129b35c231d086f9e69522d1691162ad"
Oct 31 04:00:16.646000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.645981    5831 scope.go:122] "RemoveContainer" containerID="e9d31fbae0b2b7841eeacfb202a06ac3a862cf17d4b0c083ffa0a99ef87153c6"
Oct 31 04:00:16.646360 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:16.646336    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj_odh-ai-gateway-infra(0245e787-0d08-4c8b-8af5-86ef5b363e6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f"
Oct 31 04:00:16.647699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.647678    5831 generic.go:356] "Generic (PLEG): container finished" podID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerID="294b73fc015fa1c24de3e50be44d733e0d8807329971f376526ac98d44a1c623" exitCode=60
Oct 31 04:00:16.647815 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.647755    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerDied","Data":"294b73fc015fa1c24de3e50be44d733e0d8807329971f376526ac98d44a1c623"}
Oct 31 04:00:16.648125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.648107    5831 scope.go:122] "RemoveContainer" containerID="294b73fc015fa1c24de3e50be44d733e0d8807329971f376526ac98d44a1c623"
Oct 31 04:00:16.648422 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:16.648403    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq_odh-ai-gateway-infra(8a82ece1-0234-4fe0-a827-a1fa6881319b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b"
Oct 31 04:00:16.649826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.649805    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerStarted","Data":"143cf5c1bfa73a05c1ac5287b29424e78e75adebac96d5374b570c6d98a639e1"}
Oct 31 04:00:16.662730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.662175    5831 scope.go:122] "RemoveContainer" containerID="24ac4e076b49b1bd8acd22cc6659c0c3c80065f21e1a94342ca73d41e27ce6a6"
Oct 31 04:00:16.751241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.751148    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:16.869814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:16.869776    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:17.128520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.128423    5831 scope.go:122] "RemoveContainer" containerID="45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"
Oct 31 04:00:17.129038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.128570    5831 scope.go:122] "RemoveContainer" containerID="d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"
Oct 31 04:00:17.659563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.659463    5831 generic.go:356] "Generic (PLEG): container finished" podID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerID="143cf5c1bfa73a05c1ac5287b29424e78e75adebac96d5374b570c6d98a639e1" exitCode=60
Oct 31 04:00:17.659563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.659554    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerDied","Data":"143cf5c1bfa73a05c1ac5287b29424e78e75adebac96d5374b570c6d98a639e1"}
Oct 31 04:00:17.659795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.659618    5831 scope.go:122] "RemoveContainer" containerID="7cd345e915b19837b20b4132f7f35013fb01234adccc1842b7b7e340c4be792e"
Oct 31 04:00:17.660138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.660113    5831 scope.go:122] "RemoveContainer" containerID="143cf5c1bfa73a05c1ac5287b29424e78e75adebac96d5374b570c6d98a639e1"
Oct 31 04:00:17.660482 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:17.660459    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774_odh-ai-gateway-infra(51a59bad-9153-4c9a-bb71-17e3f0e20567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567"
Oct 31 04:00:17.665554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.665512    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerStarted","Data":"4fe95aa91410ab70f39dacfe7334e1e4c64bddcba0310c45ae00bcad4bfe9bad"}
Oct 31 04:00:17.847607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.847580    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:17.896183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.896154    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:17.976209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:17.976162    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:18.001719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.001678    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\") pod \"8a82ece1-0234-4fe0-a827-a1fa6881319b\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") "
Oct 31 04:00:18.001932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.001740    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\") pod \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") "
Oct 31 04:00:18.001932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.001824    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\" (UniqueName: \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\") pod \"8a82ece1-0234-4fe0-a827-a1fa6881319b\" (UID: \"8a82ece1-0234-4fe0-a827-a1fa6881319b\") "
Oct 31 04:00:18.001932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.001923    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\" (UniqueName: \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\") pod \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\" (UID: \"0245e787-0d08-4c8b-8af5-86ef5b363e6f\") "
Oct 31 04:00:18.002180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.002157    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca" pod "0245e787-0d08-4c8b-8af5-86ef5b363e6f" (UID: "0245e787-0d08-4c8b-8af5-86ef5b363e6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.002180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.002162    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca" pod "8a82ece1-0234-4fe0-a827-a1fa6881319b" (UID: "8a82ece1-0234-4fe0-a827-a1fa6881319b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.002479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.002461    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82ece1-0234-4fe0-a827-a1fa6881319b-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.002479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.002480    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0245e787-0d08-4c8b-8af5-86ef5b363e6f-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.004276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.004248    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88" pod "8a82ece1-0234-4fe0-a827-a1fa6881319b" (UID: "8a82ece1-0234-4fe0-a827-a1fa6881319b"). InnerVolumeSpecName "kube-api-access-zjc88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.004673 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.004654    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl" pod "0245e787-0d08-4c8b-8af5-86ef5b363e6f" (UID: "0245e787-0d08-4c8b-8af5-86ef5b363e6f"). InnerVolumeSpecName "kube-api-access-q4bjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.104453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.104407    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjc88\" (UniqueName: \"kubernetes.io/projected/8a82ece1-0234-4fe0-a827-a1fa6881319b-kube-api-access-zjc88\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.104453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.104449    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4bjl\" (UniqueName: \"kubernetes.io/projected/0245e787-0d08-4c8b-8af5-86ef5b363e6f-kube-api-access-q4bjl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.674421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.674344    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"
Oct 31 04:00:18.674421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.674384    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj" event={"ID":"0245e787-0d08-4c8b-8af5-86ef5b363e6f","Type":"ContainerDied","Data":"515f8e0824c22801d8ce97b91e837e32888776b089163679e471ec22fca53da8"}
Oct 31 04:00:18.674987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.674450    5831 scope.go:122] "RemoveContainer" containerID="e9d31fbae0b2b7841eeacfb202a06ac3a862cf17d4b0c083ffa0a99ef87153c6"
Oct 31 04:00:18.676271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.676240    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq" event={"ID":"8a82ece1-0234-4fe0-a827-a1fa6881319b","Type":"ContainerDied","Data":"a4084387d7014ef9fcef3adf611c37a163ce0b705563e3b6c84dc81110bf8f7f"}
Oct 31 04:00:18.676403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.676249    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"
Oct 31 04:00:18.678643 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.678614    5831 generic.go:356] "Generic (PLEG): container finished" podID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerID="4fe95aa91410ab70f39dacfe7334e1e4c64bddcba0310c45ae00bcad4bfe9bad" exitCode=60
Oct 31 04:00:18.678764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.678695    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerDied","Data":"4fe95aa91410ab70f39dacfe7334e1e4c64bddcba0310c45ae00bcad4bfe9bad"}
Oct 31 04:00:18.679132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.679114    5831 scope.go:122] "RemoveContainer" containerID="4fe95aa91410ab70f39dacfe7334e1e4c64bddcba0310c45ae00bcad4bfe9bad"
Oct 31 04:00:18.679385 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:18.679364    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s_odh-ai-gateway-infra(57f77ad0-4af7-46eb-a782-76110ea378bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf"
Oct 31 04:00:18.680839 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.680819    5831 generic.go:356] "Generic (PLEG): container finished" podID="e89ad831-bed1-474d-a852-c8230aba7eac" containerID="d413e2538cbd1858bc30b1641645ddc6739cd9ff4d4bec4b477af4db56738df7" exitCode=60
Oct 31 04:00:18.680933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.680882    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerDied","Data":"d413e2538cbd1858bc30b1641645ddc6739cd9ff4d4bec4b477af4db56738df7"}
Oct 31 04:00:18.681234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.681217    5831 scope.go:122] "RemoveContainer" containerID="d413e2538cbd1858bc30b1641645ddc6739cd9ff4d4bec4b477af4db56738df7"
Oct 31 04:00:18.681475 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:18.681458    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd_odh-ai-gateway-infra(e89ad831-bed1-474d-a852-c8230aba7eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" podUID="e89ad831-bed1-474d-a852-c8230aba7eac"
Oct 31 04:00:18.687101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.687086    5831 scope.go:122] "RemoveContainer" containerID="294b73fc015fa1c24de3e50be44d733e0d8807329971f376526ac98d44a1c623"
Oct 31 04:00:18.697619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.697585    5831 scope.go:122] "RemoveContainer" containerID="d8abe2e9c0d162033a1d9131ffd7df40e0d21ded8464227dd2437819a147e2aa"
Oct 31 04:00:18.712524 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.712497    5831 scope.go:122] "RemoveContainer" containerID="45f0d9411bbab7c46efbd3865045637abe8c697a91956ad56a3381769152500c"
Oct 31 04:00:18.859576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.859548    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:18.898442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.898406    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:18.915638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.915609    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\" (UniqueName: \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\") pod \"51a59bad-9153-4c9a-bb71-17e3f0e20567\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") "
Oct 31 04:00:18.915840 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.915742    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\") pod \"51a59bad-9153-4c9a-bb71-17e3f0e20567\" (UID: \"51a59bad-9153-4c9a-bb71-17e3f0e20567\") "
Oct 31 04:00:18.916193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.916159    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca" pod "51a59bad-9153-4c9a-bb71-17e3f0e20567" (UID: "51a59bad-9153-4c9a-bb71-17e3f0e20567"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.918097 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.918068    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg" pod "51a59bad-9153-4c9a-bb71-17e3f0e20567" (UID: "51a59bad-9153-4c9a-bb71-17e3f0e20567"). InnerVolumeSpecName "kube-api-access-27mcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.939363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.939327    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:18.944026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:18.944000    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890320-d5rzq"]
Oct 31 04:00:19.017296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.017251    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mcg\" (UniqueName: \"kubernetes.io/projected/51a59bad-9153-4c9a-bb71-17e3f0e20567-kube-api-access-27mcg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.017296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.017292    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a59bad-9153-4c9a-bb71-17e3f0e20567-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.158289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.158255    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:19.163621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.163581    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890320-zdlmj"]
Oct 31 04:00:19.698473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.698436    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774" event={"ID":"51a59bad-9153-4c9a-bb71-17e3f0e20567","Type":"ContainerDied","Data":"740c69a21c766f53f59e2de6761fdafa4433703942b81d6621d681ff568077cc"}
Oct 31 04:00:19.698924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.698559    5831 scope.go:122] "RemoveContainer" containerID="143cf5c1bfa73a05c1ac5287b29424e78e75adebac96d5374b570c6d98a639e1"
Oct 31 04:00:19.698924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.698447    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"
Oct 31 04:00:19.871879 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.871853    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:19.908987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.908948    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:19.910423 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.910392    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890320-8n774"]
Oct 31 04:00:19.928794 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.928763    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\") pod \"57f77ad0-4af7-46eb-a782-76110ea378bf\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") "
Oct 31 04:00:19.928947 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.928881    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\" (UniqueName: \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\") pod \"57f77ad0-4af7-46eb-a782-76110ea378bf\" (UID: \"57f77ad0-4af7-46eb-a782-76110ea378bf\") "
Oct 31 04:00:19.929119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.929096    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca" pod "57f77ad0-4af7-46eb-a782-76110ea378bf" (UID: "57f77ad0-4af7-46eb-a782-76110ea378bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:19.929315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.929301    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f77ad0-4af7-46eb-a782-76110ea378bf-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.931191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:19.931162    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt" pod "57f77ad0-4af7-46eb-a782-76110ea378bf" (UID: "57f77ad0-4af7-46eb-a782-76110ea378bf"). InnerVolumeSpecName "kube-api-access-8fwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:20.030142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.030056    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwjt\" (UniqueName: \"kubernetes.io/projected/57f77ad0-4af7-46eb-a782-76110ea378bf-kube-api-access-8fwjt\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.141979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.141938    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" path="/var/lib/kubelet/pods/0245e787-0d08-4c8b-8af5-86ef5b363e6f/volumes"
Oct 31 04:00:20.142614 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.142588    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" path="/var/lib/kubelet/pods/51a59bad-9153-4c9a-bb71-17e3f0e20567/volumes"
Oct 31 04:00:20.143182 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.143160    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" path="/var/lib/kubelet/pods/8a82ece1-0234-4fe0-a827-a1fa6881319b/volumes"
Oct 31 04:00:20.278256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.278223    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:20.458889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.458866    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:20.536234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.536201    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\") pod \"e89ad831-bed1-474d-a852-c8230aba7eac\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") "
Oct 31 04:00:20.536403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.536247    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\" (UniqueName: \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\") pod \"e89ad831-bed1-474d-a852-c8230aba7eac\" (UID: \"e89ad831-bed1-474d-a852-c8230aba7eac\") "
Oct 31 04:00:20.536621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.536599    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca" pod "e89ad831-bed1-474d-a852-c8230aba7eac" (UID: "e89ad831-bed1-474d-a852-c8230aba7eac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:20.538596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.538576    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h" pod "e89ad831-bed1-474d-a852-c8230aba7eac" (UID: "e89ad831-bed1-474d-a852-c8230aba7eac"). InnerVolumeSpecName "kube-api-access-jfw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:20.637359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.637282    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad831-bed1-474d-a852-c8230aba7eac-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.637359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.637310    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfw5h\" (UniqueName: \"kubernetes.io/projected/e89ad831-bed1-474d-a852-c8230aba7eac-kube-api-access-jfw5h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.711034 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.711000    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s" event={"ID":"57f77ad0-4af7-46eb-a782-76110ea378bf","Type":"ContainerDied","Data":"55063738592ffe883f2b6d100840741d6ed73cdb7abbcf6994aa8d8f2ef71bbe"}
Oct 31 04:00:20.711561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.711068    5831 scope.go:122] "RemoveContainer" containerID="4fe95aa91410ab70f39dacfe7334e1e4c64bddcba0310c45ae00bcad4bfe9bad"
Oct 31 04:00:20.711561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.711014    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"
Oct 31 04:00:20.713083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.713065    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"
Oct 31 04:00:20.713205 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.713110    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd" event={"ID":"e89ad831-bed1-474d-a852-c8230aba7eac","Type":"ContainerDied","Data":"7481a48c85f65d5880bc57ed0563118e3bd80729ee8470dbd62bd24cc9258394"}
Oct 31 04:00:20.724544 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.724509    5831 scope.go:122] "RemoveContainer" containerID="d413e2538cbd1858bc30b1641645ddc6739cd9ff4d4bec4b477af4db56738df7"
Oct 31 04:00:20.864086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.864048    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:20.870347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.870315    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-502ec2-29890320-7tnzd"]
Oct 31 04:00:20.973930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.973885    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:20.977142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:20.977120    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd5b85-29890320-rdh6s"]
Oct 31 04:00:22.136805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:22.136771    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" path="/var/lib/kubelet/pods/57f77ad0-4af7-46eb-a782-76110ea378bf/volumes"
Oct 31 04:00:22.137268 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:22.137252    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" path="/var/lib/kubelet/pods/e89ad831-bed1-474d-a852-c8230aba7eac/volumes"
Oct 31 04:00:42.978851 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:00:42.978810    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:00:55.923345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:00:55.923305    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 04:01:06.663987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:06.663945    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:01:10.946488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:10.946438    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" podUID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" containerName="main" containerID="cri-o://6ed597fdb0dcb51ca5a7befd19ba430f636699d35ea3548cf44a79227baffaac" gracePeriod=45
Oct 31 04:01:11.193135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.193104    5831 generic.go:356] "Generic (PLEG): container finished" podID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" containerID="6ed597fdb0dcb51ca5a7befd19ba430f636699d35ea3548cf44a79227baffaac" exitCode=0
Oct 31 04:01:11.193449 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.193176    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" event={"ID":"52550c21-6da5-4a9b-81ac-e00b9e8d1805","Type":"ContainerDied","Data":"6ed597fdb0dcb51ca5a7befd19ba430f636699d35ea3548cf44a79227baffaac"}
Oct 31 04:01:11.305554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.305056    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 04:01:11.477050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477019    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\" (UniqueName: \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477262 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477116    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477262 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477235    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477296    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477327    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477364    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:11.477401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477390    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\" (UniqueName: \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\") pod \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\" (UID: \"52550c21-6da5-4a9b-81ac-e00b9e8d1805\") "
Oct 31 04:01:11.477674 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477583    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:11.477739 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.477665    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:11.478188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.478166    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:11.478289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.478192    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:11.478289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.478221    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:11.479649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.479622    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "kube-api-access-c4lxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:11.479906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.479677    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:11.479906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.479795    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs" pod "52550c21-6da5-4a9b-81ac-e00b9e8d1805" (UID: "52550c21-6da5-4a9b-81ac-e00b9e8d1805"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:11.579303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.579266    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52550c21-6da5-4a9b-81ac-e00b9e8d1805-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:11.579303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.579299    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52550c21-6da5-4a9b-81ac-e00b9e8d1805-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:11.579303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:11.579309    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lxm\" (UniqueName: \"kubernetes.io/projected/52550c21-6da5-4a9b-81ac-e00b9e8d1805-kube-api-access-c4lxm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:12.202138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:12.202103    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc" event={"ID":"52550c21-6da5-4a9b-81ac-e00b9e8d1805","Type":"ContainerDied","Data":"54322ee8fa3f2ca8ae9bce20da5784fa2f84fb0b6e7c67a9b23de96829e44035"}
Oct 31 04:01:12.202707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:12.202156    5831 scope.go:122] "RemoveContainer" containerID="6ed597fdb0dcb51ca5a7befd19ba430f636699d35ea3548cf44a79227baffaac"
Oct 31 04:01:12.202707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:12.202109    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"
Oct 31 04:01:12.357022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:12.356983    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 04:01:12.360180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:12.360158    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-ab9b62/praxis-route-ab9b62-kserve-546dc998d5-w5lwc"]
Oct 31 04:01:12.980802 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:01:12.980768    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:01:14.167382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:14.167348    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" path="/var/lib/kubelet/pods/52550c21-6da5-4a9b-81ac-e00b9e8d1805/volumes"
Oct 31 04:01:19.500373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:19.500341    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 04:01:32.746990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.746904    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"]
Oct 31 04:01:32.747915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747892    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747919    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747937    5831 cpu_manager.go:414] "containerMap: removing container" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747946    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747960    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747968    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.747980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747979    5831 cpu_manager.go:414] "containerMap: removing container" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.747987    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748014    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748023    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748045    5831 cpu_manager.go:414] "containerMap: removing container" podUID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" containerName="main"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748052    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" containerName="main"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748062    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748069    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748078    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748086    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748095    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748102    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748112    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748119    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748136    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748142    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748155    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748162    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748199    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748207    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748375    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748388    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748398    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748407    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748418    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748434    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748445    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748457    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748466    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748477    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748486    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748499    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="52550c21-6da5-4a9b-81ac-e00b9e8d1805" containerName="main"
Oct 31 04:01:32.748782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.748509    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82ece1-0234-4fe0-a827-a1fa6881319b" containerName="cleanup"
Oct 31 04:01:32.753108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.753088    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.801842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.801804    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"]
Oct 31 04:01:32.817363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.817322    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.817783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.817755    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.818004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.817986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnb4p\" (UniqueName: \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.842711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.842680    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:32.843119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.843098    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:01:32.920856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.919720    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.920856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.919891    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.920856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.919968    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnb4p\" (UniqueName: \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.922323 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.922290    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.924661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.924637    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:32.975639 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:32.975602    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnb4p\" (UniqueName: \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\") pod \"maas-api-revoke-sub-fd3013f4-bn76n\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:33.071833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:33.071742    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:33.457564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:33.457519    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"]
Oct 31 04:01:33.459312 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:01:33.459283    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba36bab_d797_40d0_b8d7_9f0436fd2e2e.slice/crio-29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a WatchSource:0}: Error finding container 29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a: Status 404 returned error can't find the container with id 29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a
Oct 31 04:01:34.408224 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.408191    5831 generic.go:356] "Generic (PLEG): container finished" podID="aba36bab-d797-40d0-b8d7-9f0436fd2e2e" containerID="b116961beb5c036fdd9ec08b561f14dd1b37aaf1f05227724a7e79fe1c732910" exitCode=0
Oct 31 04:01:34.408639 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.408314    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n" event={"ID":"aba36bab-d797-40d0-b8d7-9f0436fd2e2e","Type":"ContainerDied","Data":"b116961beb5c036fdd9ec08b561f14dd1b37aaf1f05227724a7e79fe1c732910"}
Oct 31 04:01:34.408639 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.408364    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n" event={"ID":"aba36bab-d797-40d0-b8d7-9f0436fd2e2e","Type":"ContainerStarted","Data":"29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a"}
Oct 31 04:01:34.522336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.522284    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerName="main" containerID="cri-o://b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115" gracePeriod=45
Oct 31 04:01:34.562857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.562825    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 04:01:34.850283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.850252    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 04:01:34.971420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971383    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.971751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971707    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.971751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971731    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:34.971924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971776    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\" (UniqueName: \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.971924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971812    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.971924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971875    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.971924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971905    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\") pod \"fd85d78e-4211-4de9-b36d-93d48725f9bb\" (UID: \"fd85d78e-4211-4de9-b36d-93d48725f9bb\") "
Oct 31 04:01:34.972144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.971936    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:34.972144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.972122    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:34.972545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.972468    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:34.972545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.972494    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:34.972545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.972510    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:34.977692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.977663    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:34.977692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.977672    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:34.977898 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:34.977698    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh" pod "fd85d78e-4211-4de9-b36d-93d48725f9bb" (UID: "fd85d78e-4211-4de9-b36d-93d48725f9bb"). InnerVolumeSpecName "kube-api-access-mt8jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:35.073315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.073275    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8jh\" (UniqueName: \"kubernetes.io/projected/fd85d78e-4211-4de9-b36d-93d48725f9bb-kube-api-access-mt8jh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:35.073315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.073303    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd85d78e-4211-4de9-b36d-93d48725f9bb-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:35.073315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.073313    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd85d78e-4211-4de9-b36d-93d48725f9bb-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:35.424604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.424498    5831 generic.go:356] "Generic (PLEG): container finished" podID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerID="b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115" exitCode=0
Oct 31 04:01:35.425104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.424993    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"
Oct 31 04:01:35.425924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.425774    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" event={"ID":"fd85d78e-4211-4de9-b36d-93d48725f9bb","Type":"ContainerDied","Data":"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"}
Oct 31 04:01:35.425924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.425817    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" event={"ID":"fd85d78e-4211-4de9-b36d-93d48725f9bb","Type":"ContainerDied","Data":"0c5e2df7e3808f03e08edae006bf103962440bef0716eda358213a2566df87cc"}
Oct 31 04:01:35.425924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.425839    5831 scope.go:122] "RemoveContainer" containerID="b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"
Oct 31 04:01:35.440342 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.440316    5831 scope.go:122] "RemoveContainer" containerID="b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"
Oct 31 04:01:35.440720 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:01:35.440695    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115\": container with ID starting with b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115 not found: ID does not exist" containerID="b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"
Oct 31 04:01:35.440822 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.440733    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115"} err="failed to get container status \"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115\": rpc error: code = NotFound desc = could not find container \"b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115\": container with ID starting with b972e59ccc9d4e6f04e9bc7d437e5f420ae58a4783b7d547274deb20d8033115 not found: ID does not exist"
Oct 31 04:01:35.582986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.582927    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 04:01:35.589544 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.589493    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj"]
Oct 31 04:01:35.605646 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.605612    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:35.684033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.683997    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\" (UniqueName: \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\") pod \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") "
Oct 31 04:01:35.684215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.684137    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\") pod \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") "
Oct 31 04:01:35.684326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.684263    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\") pod \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\" (UID: \"aba36bab-d797-40d0-b8d7-9f0436fd2e2e\") "
Oct 31 04:01:35.684696 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.684471    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca" pod "aba36bab-d797-40d0-b8d7-9f0436fd2e2e" (UID: "aba36bab-d797-40d0-b8d7-9f0436fd2e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:35.684771 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.684754    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:35.686685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.686654    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p" pod "aba36bab-d797-40d0-b8d7-9f0436fd2e2e" (UID: "aba36bab-d797-40d0-b8d7-9f0436fd2e2e"). InnerVolumeSpecName "kube-api-access-fnb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:35.686924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.686899    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca" pod "aba36bab-d797-40d0-b8d7-9f0436fd2e2e" (UID: "aba36bab-d797-40d0-b8d7-9f0436fd2e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:35.743160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.743113    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-502ec2/auth-test-model-502ec2-kserve-69f958db9c-8wckj" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerName="main" probeResult="failure" output="Get \"https://10.132.1.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:01:35.785791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.785751    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:35.785791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:35.785785    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnb4p\" (UniqueName: \"kubernetes.io/projected/aba36bab-d797-40d0-b8d7-9f0436fd2e2e-kube-api-access-fnb4p\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.146274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.146186    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" path="/var/lib/kubelet/pods/fd85d78e-4211-4de9-b36d-93d48725f9bb/volumes"
Oct 31 04:01:36.435104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.435074    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"
Oct 31 04:01:36.435620 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.435167    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n" event={"ID":"aba36bab-d797-40d0-b8d7-9f0436fd2e2e","Type":"ContainerDied","Data":"29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a"}
Oct 31 04:01:36.435620 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.435257    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ece5a14ecf224f19313a082375a1cbf279112e14a408153e84be1c7ffbad4a"
Oct 31 04:01:36.581373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.581331    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"]
Oct 31 04:01:36.582096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582074    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:36.582096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582097    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582120    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582127    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89ad831-bed1-474d-a852-c8230aba7eac" containerName="cleanup"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582142    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerName="main"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582149    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerName="main"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582197    5831 cpu_manager.go:414] "containerMap: removing container" podUID="aba36bab-d797-40d0-b8d7-9f0436fd2e2e" containerName="revoke-keys"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582204    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba36bab-d797-40d0-b8d7-9f0436fd2e2e" containerName="revoke-keys"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582220    5831 cpu_manager.go:414] "containerMap: removing container" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:36.582298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582226    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:36.582749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582380    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd85d78e-4211-4de9-b36d-93d48725f9bb" containerName="main"
Oct 31 04:01:36.582749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582396    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a59bad-9153-4c9a-bb71-17e3f0e20567" containerName="cleanup"
Oct 31 04:01:36.582749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582410    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba36bab-d797-40d0-b8d7-9f0436fd2e2e" containerName="revoke-keys"
Oct 31 04:01:36.582749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582418    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f77ad0-4af7-46eb-a782-76110ea378bf" containerName="cleanup"
Oct 31 04:01:36.582749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.582435    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0245e787-0d08-4c8b-8af5-86ef5b363e6f" containerName="cleanup"
Oct 31 04:01:36.587484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.587463    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.636176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.636148    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:01:36.636360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.636215    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:36.638933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.638908    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"]
Oct 31 04:01:36.716648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.716519    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.716825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.716663    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ztb\" (UniqueName: \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.716825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.716760    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.817793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.817760    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ztb\" (UniqueName: \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.818011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.817813    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.818011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.817957    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.818739 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.818661    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.820480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.820447    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.870892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.870856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ztb\" (UniqueName: \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\") pod \"maas-api-revoke-sub-969f24a4-q7w76\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:36.901611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:36.901574    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:37.278400 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:37.278376    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"]
Oct 31 04:01:37.280579 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:01:37.280511    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788fde44_7590_4864_8392_0f072d468bd4.slice/crio-dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55 WatchSource:0}: Error finding container dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55: Status 404 returned error can't find the container with id dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55
Oct 31 04:01:37.444030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:37.443997    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76" event={"ID":"788fde44-7590-4864-8392-0f072d468bd4","Type":"ContainerStarted","Data":"dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55"}
Oct 31 04:01:38.452791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:38.452759    5831 generic.go:356] "Generic (PLEG): container finished" podID="788fde44-7590-4864-8392-0f072d468bd4" containerID="271abe3623dab91b49a301a748ae060bbb8c0bfe25130c56e00eac93cb20f271" exitCode=0
Oct 31 04:01:38.453195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:38.452844    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76" event={"ID":"788fde44-7590-4864-8392-0f072d468bd4","Type":"ContainerDied","Data":"271abe3623dab91b49a301a748ae060bbb8c0bfe25130c56e00eac93cb20f271"}
Oct 31 04:01:39.563767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.563724    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing" containerID="cri-o://b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2" gracePeriod=25
Oct 31 04:01:39.623519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.623488    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:39.754503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.754420    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\" (UniqueName: \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\") pod \"788fde44-7590-4864-8392-0f072d468bd4\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") "
Oct 31 04:01:39.754503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.754478    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\") pod \"788fde44-7590-4864-8392-0f072d468bd4\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") "
Oct 31 04:01:39.754778 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.754648    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\") pod \"788fde44-7590-4864-8392-0f072d468bd4\" (UID: \"788fde44-7590-4864-8392-0f072d468bd4\") "
Oct 31 04:01:39.754951 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.754921    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca" pod "788fde44-7590-4864-8392-0f072d468bd4" (UID: "788fde44-7590-4864-8392-0f072d468bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:39.755313 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.755257    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788fde44-7590-4864-8392-0f072d468bd4-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:39.756912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.756887    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb" pod "788fde44-7590-4864-8392-0f072d468bd4" (UID: "788fde44-7590-4864-8392-0f072d468bd4"). InnerVolumeSpecName "kube-api-access-j2ztb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:39.757031 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.757008    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca" pod "788fde44-7590-4864-8392-0f072d468bd4" (UID: "788fde44-7590-4864-8392-0f072d468bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:39.856173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.856132    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788fde44-7590-4864-8392-0f072d468bd4-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:39.856173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:39.856166    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ztb\" (UniqueName: \"kubernetes.io/projected/788fde44-7590-4864-8392-0f072d468bd4-kube-api-access-j2ztb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.469682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:40.469647    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"
Oct 31 04:01:40.469682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:40.469655    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76" event={"ID":"788fde44-7590-4864-8392-0f072d468bd4","Type":"ContainerDied","Data":"dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55"}
Oct 31 04:01:40.469972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:40.469699    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa54e30c50dca1a783fa6d5ad278f7d922059dadad5bffbc5b3f8c1caa87f55"
Oct 31 04:01:42.984109 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:01:42.984070    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:01:45.961747 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:45.961703    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:01:47.339106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.339066    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"]
Oct 31 04:01:47.339828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.339809    5831 cpu_manager.go:414] "containerMap: removing container" podUID="788fde44-7590-4864-8392-0f072d468bd4" containerName="revoke-keys"
Oct 31 04:01:47.339919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.339829    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788fde44-7590-4864-8392-0f072d468bd4" containerName="revoke-keys"
Oct 31 04:01:47.339970 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.339961    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="788fde44-7590-4864-8392-0f072d468bd4" containerName="revoke-keys"
Oct 31 04:01:47.343823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.343799    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.366034 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.366003    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmrd\" (UniqueName: \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.366165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.366141    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.366306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.366285    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.436936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.436905    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:47.436936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.436939    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:01:47.442617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.442584    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"]
Oct 31 04:01:47.467492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.467445    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.467703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.467596    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmrd\" (UniqueName: \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.467703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.467687    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.468189 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.468164    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.470767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.470741    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.530118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.530090    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmrd\" (UniqueName: \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\") pod \"maas-api-revoke-sub-2d1053b9-khjbd\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:47.658373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:47.658298    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:48.038223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:48.038195    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"]
Oct 31 04:01:48.039732 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:01:48.039702    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03184725_b9aa_4be1_be70_c1519ad3b79a.slice/crio-8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d WatchSource:0}: Error finding container 8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d: Status 404 returned error can't find the container with id 8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d
Oct 31 04:01:48.543431 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:48.543342    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd" event={"ID":"03184725-b9aa-4be1-be70-c1519ad3b79a","Type":"ContainerStarted","Data":"80be0a736687bd06d15a390acc20ba731003deaf769c936c87a0fe1ad2c3cf8d"}
Oct 31 04:01:48.543431 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:48.543403    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd" event={"ID":"03184725-b9aa-4be1-be70-c1519ad3b79a","Type":"ContainerStarted","Data":"8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d"}
Oct 31 04:01:48.651021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:48.650966    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd" podStartSLOduration=1.650952149 podStartE2EDuration="1.650952149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:48.648656648 +0000 UTC m=+3258.663218357" watchObservedRunningTime="2026-10-31 04:01:48.650952149 +0000 UTC m=+3258.665513857"
Oct 31 04:01:49.551912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:49.551882    5831 generic.go:356] "Generic (PLEG): container finished" podID="03184725-b9aa-4be1-be70-c1519ad3b79a" containerID="80be0a736687bd06d15a390acc20ba731003deaf769c936c87a0fe1ad2c3cf8d" exitCode=0
Oct 31 04:01:49.552349 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:49.551972    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd" event={"ID":"03184725-b9aa-4be1-be70-c1519ad3b79a","Type":"ContainerDied","Data":"80be0a736687bd06d15a390acc20ba731003deaf769c936c87a0fe1ad2c3cf8d"}
Oct 31 04:01:50.788636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.788605    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:50.940251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.936034    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\" (UniqueName: \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\") pod \"03184725-b9aa-4be1-be70-c1519ad3b79a\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") "
Oct 31 04:01:50.940251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.936159    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\") pod \"03184725-b9aa-4be1-be70-c1519ad3b79a\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") "
Oct 31 04:01:50.940251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.936232    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\") pod \"03184725-b9aa-4be1-be70-c1519ad3b79a\" (UID: \"03184725-b9aa-4be1-be70-c1519ad3b79a\") "
Oct 31 04:01:50.943663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.943601    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd" pod "03184725-b9aa-4be1-be70-c1519ad3b79a" (UID: "03184725-b9aa-4be1-be70-c1519ad3b79a"). InnerVolumeSpecName "kube-api-access-wvmrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:50.948178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.947717    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca" pod "03184725-b9aa-4be1-be70-c1519ad3b79a" (UID: "03184725-b9aa-4be1-be70-c1519ad3b79a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:50.949965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:50.949933    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca" pod "03184725-b9aa-4be1-be70-c1519ad3b79a" (UID: "03184725-b9aa-4be1-be70-c1519ad3b79a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:51.038958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.038922    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmrd\" (UniqueName: \"kubernetes.io/projected/03184725-b9aa-4be1-be70-c1519ad3b79a-kube-api-access-wvmrd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:51.038958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.038953    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:51.039169 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.038969    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03184725-b9aa-4be1-be70-c1519ad3b79a-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:51.569204 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.569172    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"
Oct 31 04:01:51.569395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.569204    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd" event={"ID":"03184725-b9aa-4be1-be70-c1519ad3b79a","Type":"ContainerDied","Data":"8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d"}
Oct 31 04:01:51.569395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:51.569280    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8549886d2e866d88b9d979aa4b1b0cb6a32c63207a784eb58d6e4d0c05fe738d"
Oct 31 04:01:55.961731 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:55.961687    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:01:59.850072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:59.850040    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 04:01:59.949813 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:59.949775    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\" (UniqueName: \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\") pod \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") "
Oct 31 04:01:59.950021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:59.949847    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\") pod \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\" (UID: \"7565c1b3-2899-44b4-981b-bf9d2e8a06d8\") "
Oct 31 04:01:59.950274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:59.950250    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume" pod "7565c1b3-2899-44b4-981b-bf9d2e8a06d8" (UID: "7565c1b3-2899-44b4-981b-bf9d2e8a06d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:59.952315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:01:59.952294    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb" pod "7565c1b3-2899-44b4-981b-bf9d2e8a06d8" (UID: "7565c1b3-2899-44b4-981b-bf9d2e8a06d8"). InnerVolumeSpecName "kube-api-access-tgvtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:00.051112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.051071    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvtb\" (UniqueName: \"kubernetes.io/projected/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-kube-api-access-tgvtb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:00.051112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.051104    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7565c1b3-2899-44b4-981b-bf9d2e8a06d8-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:00.640185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.640145    5831 generic.go:356] "Generic (PLEG): container finished" podID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerID="b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2" exitCode=0
Oct 31 04:02:00.640411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.640197    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"
Oct 31 04:02:00.640411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.640232    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" event={"ID":"7565c1b3-2899-44b4-981b-bf9d2e8a06d8","Type":"ContainerDied","Data":"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"}
Oct 31 04:02:00.640411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.640300    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44" event={"ID":"7565c1b3-2899-44b4-981b-bf9d2e8a06d8","Type":"ContainerDied","Data":"36b0127ed2a686157cd6ccbf831521c51ba51343b7cf022134bb375d3ecd1d31"}
Oct 31 04:02:00.640411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.640329    5831 scope.go:122] "RemoveContainer" containerID="b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"
Oct 31 04:02:00.655682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.655660    5831 scope.go:122] "RemoveContainer" containerID="b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"
Oct 31 04:02:00.655998 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:02:00.655967    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2\": container with ID starting with b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2 not found: ID does not exist" containerID="b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"
Oct 31 04:02:00.656112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.656007    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2"} err="failed to get container status \"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2\": rpc error: code = NotFound desc = could not find container \"b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2\": container with ID starting with b597255f0c26d8675c5b33ea3e134a80e5e313f1cd8b779edd065738c6fbc3c2 not found: ID does not exist"
Oct 31 04:02:00.795663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.795623    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 04:02:00.841124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:00.841081    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ab9b62-5c6cd9644c-xnh44"]
Oct 31 04:02:02.136804 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:02.136772    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" path="/var/lib/kubelet/pods/7565c1b3-2899-44b4-981b-bf9d2e8a06d8/volumes"
Oct 31 04:02:11.129065 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:11.129024    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:02:12.984286 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:02:12.984240    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:02:17.897456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:17.897420    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 04:02:21.737862 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.737829    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"]
Oct 31 04:02:21.738471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738457    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing"
Oct 31 04:02:21.738552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738472    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing"
Oct 31 04:02:21.738552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738486    5831 cpu_manager.go:414] "containerMap: removing container" podUID="03184725-b9aa-4be1-be70-c1519ad3b79a" containerName="revoke-keys"
Oct 31 04:02:21.738552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738492    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03184725-b9aa-4be1-be70-c1519ad3b79a" containerName="revoke-keys"
Oct 31 04:02:21.738672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738635    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7565c1b3-2899-44b4-981b-bf9d2e8a06d8" containerName="payload-pre-processing"
Oct 31 04:02:21.738672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.738647    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="03184725-b9aa-4be1-be70-c1519ad3b79a" containerName="revoke-keys"
Oct 31 04:02:21.742360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.742341    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.803553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.798358    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"]
Oct 31 04:02:21.834466 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.834416    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:02:21.834703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.834621    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:02:21.884374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.884334    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.884619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.884392    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.884619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.884600    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.986263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.986223    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.986470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.986383    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.986470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.986422    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.987081 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.987056    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:21.989098 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:21.989035    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:22.039028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.038979    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:22.056655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.056616    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:22.450412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.450384    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"]
Oct 31 04:02:22.453024 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:02:22.452992    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c6cab_fbcb_484a_a861_09df39e006a2.slice/crio-0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d WatchSource:0}: Error finding container 0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d: Status 404 returned error can't find the container with id 0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d
Oct 31 04:02:22.829685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.829648    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb" event={"ID":"4f5c6cab-fbcb-484a-a861-09df39e006a2","Type":"ContainerStarted","Data":"846a09ba21f66c3ac3675a30745a0b77bff75b3affa3f9f076696d61eea4ed97"}
Oct 31 04:02:22.830197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.829706    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb" event={"ID":"4f5c6cab-fbcb-484a-a861-09df39e006a2","Type":"ContainerStarted","Data":"0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d"}
Oct 31 04:02:22.943101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:22.943047    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb" podStartSLOduration=1.943032718 podStartE2EDuration="1.943032718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:22.942393406 +0000 UTC m=+3292.956955115" watchObservedRunningTime="2026-10-31 04:02:22.943032718 +0000 UTC m=+3292.957594426"
Oct 31 04:02:23.838958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:23.838863    5831 generic.go:356] "Generic (PLEG): container finished" podID="4f5c6cab-fbcb-484a-a861-09df39e006a2" containerID="846a09ba21f66c3ac3675a30745a0b77bff75b3affa3f9f076696d61eea4ed97" exitCode=0
Oct 31 04:02:23.839341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:23.838949    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb" event={"ID":"4f5c6cab-fbcb-484a-a861-09df39e006a2","Type":"ContainerDied","Data":"846a09ba21f66c3ac3675a30745a0b77bff75b3affa3f9f076696d61eea4ed97"}
Oct 31 04:02:25.035302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.035271    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:25.236019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.235987    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\") pod \"4f5c6cab-fbcb-484a-a861-09df39e006a2\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") "
Oct 31 04:02:25.236219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.236142    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\") pod \"4f5c6cab-fbcb-484a-a861-09df39e006a2\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") "
Oct 31 04:02:25.236288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.236260    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\") pod \"4f5c6cab-fbcb-484a-a861-09df39e006a2\" (UID: \"4f5c6cab-fbcb-484a-a861-09df39e006a2\") "
Oct 31 04:02:25.236628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.236601    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca" pod "4f5c6cab-fbcb-484a-a861-09df39e006a2" (UID: "4f5c6cab-fbcb-484a-a861-09df39e006a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:25.238375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.238355    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn" pod "4f5c6cab-fbcb-484a-a861-09df39e006a2" (UID: "4f5c6cab-fbcb-484a-a861-09df39e006a2"). InnerVolumeSpecName "kube-api-access-2rsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:25.238375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.238357    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca" pod "4f5c6cab-fbcb-484a-a861-09df39e006a2" (UID: "4f5c6cab-fbcb-484a-a861-09df39e006a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:25.338006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.337968    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:25.338006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.338001    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsfn\" (UniqueName: \"kubernetes.io/projected/4f5c6cab-fbcb-484a-a861-09df39e006a2-kube-api-access-2rsfn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:25.338006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.338011    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5c6cab-fbcb-484a-a861-09df39e006a2-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:25.972155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.972131    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"
Oct 31 04:02:25.972857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.972190    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb" event={"ID":"4f5c6cab-fbcb-484a-a861-09df39e006a2","Type":"ContainerDied","Data":"0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d"}
Oct 31 04:02:25.973011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:25.972877    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a52c223a3701a8e1468dbcdd38d74b5d696a9152c02d86488e06389c7a89f5d"
Oct 31 04:02:32.919068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:32.919024    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" podUID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" containerName="main" containerID="cri-o://735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11" gracePeriod=45
Oct 31 04:02:33.193670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.193637    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 04:02:33.374040 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374006    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374050    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\" (UniqueName: \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374136    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374194    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374274    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\") pod \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\" (UID: \"8b20ebe2-66aa-409e-aff3-99d63cf54f4e\") "
Oct 31 04:02:33.374615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374358    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:33.374615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374456    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:33.374828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374741    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:33.375006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374940    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.375006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374960    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.375006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.374975    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.376946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.376916    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:33.377086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.377064    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "kube-api-access-htxml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:33.377950 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.377917    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs" pod "8b20ebe2-66aa-409e-aff3-99d63cf54f4e" (UID: "8b20ebe2-66aa-409e-aff3-99d63cf54f4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:33.476334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.476302    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.476334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.476327    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.476334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.476338    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxml\" (UniqueName: \"kubernetes.io/projected/8b20ebe2-66aa-409e-aff3-99d63cf54f4e-kube-api-access-htxml\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:33.884142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884061    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"]
Oct 31 04:02:33.884754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884737    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" containerName="main"
Oct 31 04:02:33.884830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884755    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" containerName="main"
Oct 31 04:02:33.884830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884785    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4f5c6cab-fbcb-484a-a861-09df39e006a2" containerName="revoke-keys"
Oct 31 04:02:33.884830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884790    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5c6cab-fbcb-484a-a861-09df39e006a2" containerName="revoke-keys"
Oct 31 04:02:33.884935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884898    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" containerName="main"
Oct 31 04:02:33.884935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.884910    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5c6cab-fbcb-484a-a861-09df39e006a2" containerName="revoke-keys"
Oct 31 04:02:33.888808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.888787    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:33.938445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.938410    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:02:33.980322 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.980293    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"]
Oct 31 04:02:33.982753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.982729    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:33.982878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.982858    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87tv\" (UniqueName: \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:33.982936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.982910    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:33.983894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:33.983876    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:02:34.040469 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.040428    5831 generic.go:356] "Generic (PLEG): container finished" podID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" containerID="735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11" exitCode=0
Oct 31 04:02:34.040469 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.040472    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"
Oct 31 04:02:34.040727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.040475    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" event={"ID":"8b20ebe2-66aa-409e-aff3-99d63cf54f4e","Type":"ContainerDied","Data":"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"}
Oct 31 04:02:34.040727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.040518    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv" event={"ID":"8b20ebe2-66aa-409e-aff3-99d63cf54f4e","Type":"ContainerDied","Data":"294601473c9835753ca78f82a993540c363dc84d81fa4a2818165e6303d567b0"}
Oct 31 04:02:34.040727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.040572    5831 scope.go:122] "RemoveContainer" containerID="735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"
Oct 31 04:02:34.051821 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.051801    5831 scope.go:122] "RemoveContainer" containerID="735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"
Oct 31 04:02:34.052092 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:02:34.052075    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11\": container with ID starting with 735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11 not found: ID does not exist" containerID="735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"
Oct 31 04:02:34.052179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.052101    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11"} err="failed to get container status \"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11\": rpc error: code = NotFound desc = could not find container \"735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11\": container with ID starting with 735dc48554cb0b01963d65b9e97d39fdae53672db80e51844f169f733c0a1a11 not found: ID does not exist"
Oct 31 04:02:34.084100 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.084067    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87tv\" (UniqueName: \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.084254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.084130    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.084254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.084235    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.085125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.085090    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.086766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.086744    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.174195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.174118    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87tv\" (UniqueName: \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\") pod \"maas-api-revoke-sub-d734a583-fwv5x\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.203206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.203179    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:34.227203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.227169    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 04:02:34.232830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.232797    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-93f411/auth-test-model-93f411-kserve-57d878f445-h5mdv"]
Oct 31 04:02:34.410285 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.410249    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:02:34.422144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.421220    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.467422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.467388    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:02:34.469882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.469852    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e8fa21\""
Oct 31 04:02:34.470112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.470093    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e8fa21-dockercfg-qmq7m\""
Oct 31 04:02:34.470916 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.470893    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.471047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.471031    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkq6\" (UniqueName: \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.599554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.599224    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkq6\" (UniqueName: \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.599554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.599402    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.626513 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.619173    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.642746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.642713    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:02:34.648444 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.648419    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.650110 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.650083    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:02:34.670124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.670100    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkq6\" (UniqueName: \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\") pod \"payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.683954 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.683924    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"]
Oct 31 04:02:34.684668 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:02:34.684610    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode083eb13_c637_46b0_9dc1_f9aab5bbdd72.slice/crio-77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2 WatchSource:0}: Error finding container 77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2: Status 404 returned error can't find the container with id 77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2
Oct 31 04:02:34.728088 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.728057    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqr6z\" (UniqueName: \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.728231 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.728108    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.747813 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.747775    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:34.830559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.829712    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqr6z\" (UniqueName: \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.830559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.829776    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.830559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.830467    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.883041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.883009    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqr6z\" (UniqueName: \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\") pod \"payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:34.966471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:34.966379    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:35.054579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:35.054508    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x" event={"ID":"e083eb13-c637-46b0-9dc1-f9aab5bbdd72","Type":"ContainerStarted","Data":"77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2"}
Oct 31 04:02:35.130833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:35.130799    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:02:35.131092 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:02:35.130938    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad987cc_6640_4085_85c7_8d1d2c2c3707.slice/crio-98a58c274a5061f11f77c8b63f58b15b9964897fe2e3feb1674900bd026d106b WatchSource:0}: Error finding container 98a58c274a5061f11f77c8b63f58b15b9964897fe2e3feb1674900bd026d106b: Status 404 returned error can't find the container with id 98a58c274a5061f11f77c8b63f58b15b9964897fe2e3feb1674900bd026d106b
Oct 31 04:02:35.349835 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:35.349805    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:02:35.352076 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:02:35.352051    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34eaa693_87be_4f7b_989d_c504b0e14ad1.slice/crio-412aa49d2c0b89a923310bce0cec73ab9b11fce25e167ff70a56dfe77ebaed7d WatchSource:0}: Error finding container 412aa49d2c0b89a923310bce0cec73ab9b11fce25e167ff70a56dfe77ebaed7d: Status 404 returned error can't find the container with id 412aa49d2c0b89a923310bce0cec73ab9b11fce25e167ff70a56dfe77ebaed7d
Oct 31 04:02:36.064404 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.064313    5831 generic.go:356] "Generic (PLEG): container finished" podID="e083eb13-c637-46b0-9dc1-f9aab5bbdd72" containerID="8615c03cb2418224670c7854a217e5f2164bc377a5e351f88030c2aec8591ae2" exitCode=0
Oct 31 04:02:36.064905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.064399    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x" event={"ID":"e083eb13-c637-46b0-9dc1-f9aab5bbdd72","Type":"ContainerDied","Data":"8615c03cb2418224670c7854a217e5f2164bc377a5e351f88030c2aec8591ae2"}
Oct 31 04:02:36.066288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.066261    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" event={"ID":"34eaa693-87be-4f7b-989d-c504b0e14ad1","Type":"ContainerStarted","Data":"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"}
Oct 31 04:02:36.066426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.066302    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" event={"ID":"34eaa693-87be-4f7b-989d-c504b0e14ad1","Type":"ContainerStarted","Data":"412aa49d2c0b89a923310bce0cec73ab9b11fce25e167ff70a56dfe77ebaed7d"}
Oct 31 04:02:36.066426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.066349    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:36.067992 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.067972    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" event={"ID":"4ad987cc-6640-4085-85c7-8d1d2c2c3707","Type":"ContainerStarted","Data":"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"}
Oct 31 04:02:36.068075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.068010    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" event={"ID":"4ad987cc-6640-4085-85c7-8d1d2c2c3707","Type":"ContainerStarted","Data":"98a58c274a5061f11f77c8b63f58b15b9964897fe2e3feb1674900bd026d106b"}
Oct 31 04:02:36.068075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.068035    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:36.136465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.136433    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b20ebe2-66aa-409e-aff3-99d63cf54f4e" path="/var/lib/kubelet/pods/8b20ebe2-66aa-409e-aff3-99d63cf54f4e/volumes"
Oct 31 04:02:36.276710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.276659    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" podStartSLOduration=2.276645004 podStartE2EDuration="2.276645004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:36.274127304 +0000 UTC m=+3306.288689014" watchObservedRunningTime="2026-10-31 04:02:36.276645004 +0000 UTC m=+3306.291206709"
Oct 31 04:02:36.379216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:36.379116    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" podStartSLOduration=2.379101407 podStartE2EDuration="2.379101407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:36.377926306 +0000 UTC m=+3306.392488015" watchObservedRunningTime="2026-10-31 04:02:36.379101407 +0000 UTC m=+3306.393663116"
Oct 31 04:02:37.241358 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.241331    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:37.267768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.261100    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\") pod \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") "
Oct 31 04:02:37.267768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.261212    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\") pod \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") "
Oct 31 04:02:37.267768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.261292    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\" (UniqueName: \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\") pod \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\" (UID: \"e083eb13-c637-46b0-9dc1-f9aab5bbdd72\") "
Oct 31 04:02:37.277836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.277760    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca" pod "e083eb13-c637-46b0-9dc1-f9aab5bbdd72" (UID: "e083eb13-c637-46b0-9dc1-f9aab5bbdd72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:37.280506 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.280473    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca" pod "e083eb13-c637-46b0-9dc1-f9aab5bbdd72" (UID: "e083eb13-c637-46b0-9dc1-f9aab5bbdd72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:37.281625 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.281591    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv" pod "e083eb13-c637-46b0-9dc1-f9aab5bbdd72" (UID: "e083eb13-c637-46b0-9dc1-f9aab5bbdd72"). InnerVolumeSpecName "kube-api-access-x87tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:37.363419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.363338    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:37.363419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.363368    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87tv\" (UniqueName: \"kubernetes.io/projected/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-kube-api-access-x87tv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:37.363419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:37.363383    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083eb13-c637-46b0-9dc1-f9aab5bbdd72-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.092838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.092801    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x" event={"ID":"e083eb13-c637-46b0-9dc1-f9aab5bbdd72","Type":"ContainerDied","Data":"77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2"}
Oct 31 04:02:38.092838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.092822    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"
Oct 31 04:02:38.093120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.092889    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f7158c4317ab4809199ec50590a24f781f9daad44d483bed4633aa9f9fcff2"
Oct 31 04:02:38.422901 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.422819    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:02:38.426007 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.425987    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:02:38.433914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.433894    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:02:38.436719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.436698    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:02:38.437664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.437644    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:02:38.440692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:38.440675    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:02:42.980358 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:02:42.980314    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:02:47.086139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:47.086107    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:02:47.086586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:47.086264    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:02:53.797226 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:53.797192    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"]
Oct 31 04:02:53.844681 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:53.844639    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ab9b62-drlsb"]
Oct 31 04:02:54.043549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:54.043503    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 04:02:54.044024 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:54.043824    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy" containerID="cri-o://49fa6258b5acd92249d9cc2fdd0a421b75f66051c31fa94bd6bcfe137c8f3014" gracePeriod=30
Oct 31 04:02:54.137249 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:54.137173    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5c6cab-fbcb-484a-a861-09df39e006a2" path="/var/lib/kubelet/pods/4f5c6cab-fbcb-484a-a861-09df39e006a2/volumes"
Oct 31 04:02:59.270324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.270287    5831 generic.go:356] "Generic (PLEG): container finished" podID="db69b913-b751-4968-b216-4358ea07a83f" containerID="49fa6258b5acd92249d9cc2fdd0a421b75f66051c31fa94bd6bcfe137c8f3014" exitCode=0
Oct 31 04:02:59.270852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.270356    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerDied","Data":"49fa6258b5acd92249d9cc2fdd0a421b75f66051c31fa94bd6bcfe137c8f3014"}
Oct 31 04:02:59.270852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.270467    5831 scope.go:122] "RemoveContainer" containerID="4c6aaea2480fb21d3f224f9046cb49ee5ff00f4a3c3be51e6677e19b1019e5f2"
Oct 31 04:02:59.341363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.341335    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 04:02:59.396635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396594    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.396845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396674    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.396845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396714    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.396845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396777    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.396845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396811    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.396845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396844    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.397150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396876    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.397150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.396975    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.397150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397030    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:59.397150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397071    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\") pod \"db69b913-b751-4968-b216-4358ea07a83f\" (UID: \"db69b913-b751-4968-b216-4358ea07a83f\") "
Oct 31 04:02:59.397405 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397382    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:59.397576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397474    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:59.397724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397518    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:59.397724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397605    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:59.398020 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.397994    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.398141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.398026    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.398141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.398040    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db69b913-b751-4968-b216-4358ea07a83f-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.398141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.398055    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.398141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.398070    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.399574 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.399350    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "kube-api-access-tpvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:59.399820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.399788    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:59.400128 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.400104    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:02:59.400451 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.400431    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy" pod "db69b913-b751-4968-b216-4358ea07a83f" (UID: "db69b913-b751-4968-b216-4358ea07a83f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:59.499481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.499449    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvcx\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-kube-api-access-tpvcx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.499481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.499482    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db69b913-b751-4968-b216-4358ea07a83f-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.499744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.499497    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db69b913-b751-4968-b216-4358ea07a83f-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:59.499744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:02:59.499511    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db69b913-b751-4968-b216-4358ea07a83f-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:00.279875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:00.279833    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"
Oct 31 04:03:00.279875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:00.279859    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm" event={"ID":"db69b913-b751-4968-b216-4358ea07a83f","Type":"ContainerDied","Data":"aa6f3994155e71bf21387639740bfe90fa6ecd1f6c255bd5758125dd917ea5e1"}
Oct 31 04:03:00.280459 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:00.279928    5831 scope.go:122] "RemoveContainer" containerID="49fa6258b5acd92249d9cc2fdd0a421b75f66051c31fa94bd6bcfe137c8f3014"
Oct 31 04:03:00.480568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:00.480512    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 04:03:00.526818 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:00.526778    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ab9b62-openshift-default-59c8cbd769-2lnkm"]
Oct 31 04:03:02.135927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:02.135883    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db69b913-b751-4968-b216-4358ea07a83f" path="/var/lib/kubelet/pods/db69b913-b751-4968-b216-4358ea07a83f/volumes"
Oct 31 04:03:07.510596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:07.510492    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:03:08.777562 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:08.777513    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 04:03:08.778081 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:08.777783    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" podUID="3050f374-34af-4dc7-8659-1eeb998480b2" containerName="maas-api" containerID="cri-o://a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198" gracePeriod=30
Oct 31 04:03:09.128930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.128895    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 04:03:09.258148 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.258098    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\" (UniqueName: \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\") pod \"3050f374-34af-4dc7-8659-1eeb998480b2\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") "
Oct 31 04:03:09.258348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.258158    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") pod \"3050f374-34af-4dc7-8659-1eeb998480b2\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") "
Oct 31 04:03:09.258348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.258200    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\") pod \"3050f374-34af-4dc7-8659-1eeb998480b2\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") "
Oct 31 04:03:09.258348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.258336    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\") pod \"3050f374-34af-4dc7-8659-1eeb998480b2\" (UID: \"3050f374-34af-4dc7-8659-1eeb998480b2\") "
Oct 31 04:03:09.258966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.258919    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca" pod "3050f374-34af-4dc7-8659-1eeb998480b2" (UID: "3050f374-34af-4dc7-8659-1eeb998480b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:09.259362 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.259331    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3050f374-34af-4dc7-8659-1eeb998480b2-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:09.261964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.261939    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls" pod "3050f374-34af-4dc7-8659-1eeb998480b2" (UID: "3050f374-34af-4dc7-8659-1eeb998480b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:09.263647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.263620    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx" pod "3050f374-34af-4dc7-8659-1eeb998480b2" (UID: "3050f374-34af-4dc7-8659-1eeb998480b2"). InnerVolumeSpecName "kube-api-access-b6ctx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:09.267829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.267802    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs" pod "3050f374-34af-4dc7-8659-1eeb998480b2" (UID: "3050f374-34af-4dc7-8659-1eeb998480b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:09.374945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.364444    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:09.374945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.374738    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ctx\" (UniqueName: \"kubernetes.io/projected/3050f374-34af-4dc7-8659-1eeb998480b2-kube-api-access-b6ctx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:09.374945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.374790    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3050f374-34af-4dc7-8659-1eeb998480b2-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:09.433853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.433819    5831 generic.go:356] "Generic (PLEG): container finished" podID="3050f374-34af-4dc7-8659-1eeb998480b2" containerID="a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198" exitCode=0
Oct 31 04:03:09.434041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.433882    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"
Oct 31 04:03:09.434041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.433904    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" event={"ID":"3050f374-34af-4dc7-8659-1eeb998480b2","Type":"ContainerDied","Data":"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"}
Oct 31 04:03:09.434041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.433987    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q" event={"ID":"3050f374-34af-4dc7-8659-1eeb998480b2","Type":"ContainerDied","Data":"a33f25627f2476f68f627579d040f86902c7a7ad484855cbd19c6094529fc335"}
Oct 31 04:03:09.434041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.434015    5831 scope.go:122] "RemoveContainer" containerID="a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"
Oct 31 04:03:09.448771 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.448746    5831 scope.go:122] "RemoveContainer" containerID="a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"
Oct 31 04:03:09.449134 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:09.449116    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198\": container with ID starting with a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198 not found: ID does not exist" containerID="a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"
Oct 31 04:03:09.449214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.449141    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198"} err="failed to get container status \"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198\": rpc error: code = NotFound desc = could not find container \"a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198\": container with ID starting with a1a8e047a829565135f7f1793959c55de84a03fa221ab15da6090cac7cc69198 not found: ID does not exist"
Oct 31 04:03:09.586948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.586912    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 04:03:09.592764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:09.592733    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8b85a6-55b6c49dd9-xmh7q"]
Oct 31 04:03:10.136648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:10.136617    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3050f374-34af-4dc7-8659-1eeb998480b2" path="/var/lib/kubelet/pods/3050f374-34af-4dc7-8659-1eeb998480b2/volumes"
Oct 31 04:03:12.978080 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:12.978033    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:03:14.105430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:14.105397    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 04:03:14.106123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:14.105632    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy" containerID="cri-o://3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce" gracePeriod=30
Oct 31 04:03:15.484079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.484044    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:03:15.484983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.484962    5831 cpu_manager.go:414] "containerMap: removing container" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:15.484983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.484984    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485016    5831 cpu_manager.go:414] "containerMap: removing container" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485025    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485042    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3050f374-34af-4dc7-8659-1eeb998480b2" containerName="maas-api"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485049    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3050f374-34af-4dc7-8659-1eeb998480b2" containerName="maas-api"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485073    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e083eb13-c637-46b0-9dc1-f9aab5bbdd72" containerName="revoke-keys"
Oct 31 04:03:15.485184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485080    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e083eb13-c637-46b0-9dc1-f9aab5bbdd72" containerName="revoke-keys"
Oct 31 04:03:15.485502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485254    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:15.485502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485270    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3050f374-34af-4dc7-8659-1eeb998480b2" containerName="maas-api"
Oct 31 04:03:15.485502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.485284    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e083eb13-c637-46b0-9dc1-f9aab5bbdd72" containerName="revoke-keys"
Oct 31 04:03:15.490766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.490742    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.605561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.603607    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5901fe-openshift-default-dockercfg-xxxst\""
Oct 31 04:03:15.623598 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.616619    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:03:15.686419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686381    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686440    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfd4v\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686617    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686695    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686731    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686770    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686961 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686816    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.686961 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.686948    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.687152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.687116    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788288    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788396    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788439    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788593    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788658    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788732    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788862    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788884    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788903    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfd4v\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788948    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.788987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.788979    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.789358 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.789337    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.789498 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.789473    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.791491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.791462    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.791800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.791764    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.841930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.841894    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:15.842109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:15.842067    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfd4v\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\") pod \"e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") " pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:16.107575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:16.107456    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:16.542649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:16.542618    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:03:16.543909 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:16.543885    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098782ce_3828_401e_890e_a33f83cfca62.slice/crio-c9e0794d6d76b97a000abc716785e4f872a75a169c09938b0d546955c470699c WatchSource:0}: Error finding container c9e0794d6d76b97a000abc716785e4f872a75a169c09938b0d546955c470699c: Status 404 returned error can't find the container with id c9e0794d6d76b97a000abc716785e4f872a75a169c09938b0d546955c470699c
Oct 31 04:03:16.546463 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:16.546432    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:16.546590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:16.546496    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:16.546590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:16.546560    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:17.498201 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:17.498164    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerStarted","Data":"5ad94f85d37db0df07b5ca5dff8ccd17eb62d194db5f2e9f7db905d893d2d617"}
Oct 31 04:03:17.498430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:17.498218    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerStarted","Data":"c9e0794d6d76b97a000abc716785e4f872a75a169c09938b0d546955c470699c"}
Oct 31 04:03:17.653393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:17.653227    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podStartSLOduration=2.65320766 podStartE2EDuration="2.65320766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:17.65273569 +0000 UTC m=+3347.667297396" watchObservedRunningTime="2026-10-31 04:03:17.65320766 +0000 UTC m=+3347.667769369"
Oct 31 04:03:18.108256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:18.108198    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:18.115615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:18.115589    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:18.506643 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:18.506608    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:18.507679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:18.507644    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:03:19.382781 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.382715    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 04:03:19.439005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.438969    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439018    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439061    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439096    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439138    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439162    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439259    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439284    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439320    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439443    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\") pod \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\" (UID: \"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc\") "
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439447    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:19.439648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439609    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:19.439911 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.439829    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:19.440275 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.440255    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.440320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.440282    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.440320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.440299    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.440320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.440313    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.440418 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.440326    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.441559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.441497    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:03:19.441699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.441680    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "kube-api-access-xvm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:19.442010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.441993    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:19.442062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.442031    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token" pod "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" (UID: "93e65b3a-a1fd-40bc-be30-8f1d2830d9bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:19.515163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.515129    5831 generic.go:356] "Generic (PLEG): container finished" podID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerID="3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce" exitCode=0
Oct 31 04:03:19.515340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.515190    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"
Oct 31 04:03:19.515340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.515213    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerDied","Data":"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"}
Oct 31 04:03:19.515340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.515270    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm" event={"ID":"93e65b3a-a1fd-40bc-be30-8f1d2830d9bc","Type":"ContainerDied","Data":"2873b331e1e0352b41828bfe55cd0f22ce99f4e843ffea14b792a872bae40125"}
Oct 31 04:03:19.515340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.515303    5831 scope.go:122] "RemoveContainer" containerID="3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"
Oct 31 04:03:19.527165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.527145    5831 scope.go:122] "RemoveContainer" containerID="eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"
Oct 31 04:03:19.537131 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.537112    5831 scope.go:122] "RemoveContainer" containerID="3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"
Oct 31 04:03:19.537411 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:19.537391    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce\": container with ID starting with 3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce not found: ID does not exist" containerID="3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"
Oct 31 04:03:19.537519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.537418    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce"} err="failed to get container status \"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce\": rpc error: code = NotFound desc = could not find container \"3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce\": container with ID starting with 3fffd686ebea9939975545dfc54682c12da8e8cb3c532a26ebb0b126d93d17ce not found: ID does not exist"
Oct 31 04:03:19.537519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.537441    5831 scope.go:122] "RemoveContainer" containerID="eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"
Oct 31 04:03:19.537924 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:19.537893    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397\": container with ID starting with eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397 not found: ID does not exist" containerID="eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"
Oct 31 04:03:19.538051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.537934    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397"} err="failed to get container status \"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397\": rpc error: code = NotFound desc = could not find container \"eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397\": container with ID starting with eb6ffcf240d80a774fda219552eacff981213ccadffe3e158408f26d74c3b397 not found: ID does not exist"
Oct 31 04:03:19.542823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.542794    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.543002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.542827    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvm2j\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-kube-api-access-xvm2j\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.543002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.542843    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.543002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.542858    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:19.641956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.641861    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 04:03:19.645176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:19.645150    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8b85a6-openshift-default-6754cfb9c7-zhbfm"]
Oct 31 04:03:20.136216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:20.136181    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" path="/var/lib/kubelet/pods/93e65b3a-a1fd-40bc-be30-8f1d2830d9bc/volumes"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.497800    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.498769    5831 cpu_manager.go:414] "containerMap: removing container" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.498790    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.498839    5831 cpu_manager.go:414] "containerMap: removing container" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.498847    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.499026    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.504695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.499046    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="db69b913-b751-4968-b216-4358ea07a83f" containerName="istio-proxy"
Oct 31 04:03:24.534854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.534820    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:03:24.535107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.534955    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.586795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.586765    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5901fe\""
Oct 31 04:03:24.618720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.618687    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5nb\" (UniqueName: \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.618893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.618742    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.632929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.632901    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5901fe-dockercfg-wmtss\""
Oct 31 04:03:24.636638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.636611    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:03:24.637417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.637400    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e65b3a-a1fd-40bc-be30-8f1d2830d9bc" containerName="istio-proxy"
Oct 31 04:03:24.641225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.641208    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.685607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.685577    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:03:24.720261 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.720225    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqm9\" (UniqueName: \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.720436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.720387    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.720436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.720433    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5nb\" (UniqueName: \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.720872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.720476    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.721294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.721269    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.773414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.773347    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5nb\" (UniqueName: \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\") pod \"payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.821668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.821637    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqm9\" (UniqueName: \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.821872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.821772    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.822428 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.822407    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.874494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.874459    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:24.874850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.874827    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqm9\" (UniqueName: \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\") pod \"payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:24.957814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:24.957783    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:25.252477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.252445    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:03:25.253797 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:25.253767    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbe0bca_581a_4bc4_add3_eb5c64b7f95e.slice/crio-9d258ed6b5fd724af50c42eabfec37612110e2615a7c7384ec1f4ba2af566e20 WatchSource:0}: Error finding container 9d258ed6b5fd724af50c42eabfec37612110e2615a7c7384ec1f4ba2af566e20: Status 404 returned error can't find the container with id 9d258ed6b5fd724af50c42eabfec37612110e2615a7c7384ec1f4ba2af566e20
Oct 31 04:03:25.347994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.347969    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:03:25.349816 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:25.349788    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e44c584_a865_408b_a149_fef2ad92cdac.slice/crio-ec88941846d59952ec4c9e290d10ac1a05f1278930bfc5c6f73b763ac8cffbd0 WatchSource:0}: Error finding container ec88941846d59952ec4c9e290d10ac1a05f1278930bfc5c6f73b763ac8cffbd0: Status 404 returned error can't find the container with id ec88941846d59952ec4c9e290d10ac1a05f1278930bfc5c6f73b763ac8cffbd0
Oct 31 04:03:25.585184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.585076    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" event={"ID":"5e44c584-a865-408b-a149-fef2ad92cdac","Type":"ContainerStarted","Data":"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"}
Oct 31 04:03:25.585184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.585150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" event={"ID":"5e44c584-a865-408b-a149-fef2ad92cdac","Type":"ContainerStarted","Data":"ec88941846d59952ec4c9e290d10ac1a05f1278930bfc5c6f73b763ac8cffbd0"}
Oct 31 04:03:25.585184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.585181    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:25.587083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.587050    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" event={"ID":"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e","Type":"ContainerStarted","Data":"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"}
Oct 31 04:03:25.587232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.587103    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" event={"ID":"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e","Type":"ContainerStarted","Data":"9d258ed6b5fd724af50c42eabfec37612110e2615a7c7384ec1f4ba2af566e20"}
Oct 31 04:03:25.587232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.587128    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:25.695325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.695270    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" podStartSLOduration=1.695256436 podStartE2EDuration="1.695256436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:25.692917426 +0000 UTC m=+3355.707479133" watchObservedRunningTime="2026-10-31 04:03:25.695256436 +0000 UTC m=+3355.709818135"
Oct 31 04:03:25.805217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:25.805144    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" podStartSLOduration=1.805125664 podStartE2EDuration="1.805125664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:25.800358699 +0000 UTC m=+3355.814920405" watchObservedRunningTime="2026-10-31 04:03:25.805125664 +0000 UTC m=+3355.819687372"
Oct 31 04:03:31.927333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:31.927291    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 04:03:34.015571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.015524    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:03:34.020176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.020154    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078066    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078139    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078207    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078243    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078276    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078300    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078330    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078354    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2l8\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.078555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.078396    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.183900 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.183858    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d8190b-openshift-default-dockercfg-qtpns\""
Oct 31 04:03:34.186787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186756    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186811    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186847    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186871    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186902    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186926    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2l8\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.186965    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.187079    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.187136    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.187649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.187625    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.188132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.188111    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.188206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.188165    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.188368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.188349    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.188695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.188600    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.277670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.273877    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.285749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.285671    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.288599 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.287814    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:03:34.291932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.291904    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2l8\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.295708 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.295677    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\") pod \"e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") " pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.335132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.335097    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:34.793566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.793492    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:03:34.793909 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:34.793881    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfd2d19_2fb5_4463_858a_36831814efb8.slice/crio-0d5f10cee9495e656f284be565592b4af33ce5dbd45377b87a2756cfaa71d56a WatchSource:0}: Error finding container 0d5f10cee9495e656f284be565592b4af33ce5dbd45377b87a2756cfaa71d56a: Status 404 returned error can't find the container with id 0d5f10cee9495e656f284be565592b4af33ce5dbd45377b87a2756cfaa71d56a
Oct 31 04:03:34.796748 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.796708    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:34.796861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.796794    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:34.796861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:34.796838    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:35.673820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:35.673776    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerStarted","Data":"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"}
Oct 31 04:03:35.674252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:35.673875    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerStarted","Data":"0d5f10cee9495e656f284be565592b4af33ce5dbd45377b87a2756cfaa71d56a"}
Oct 31 04:03:35.784147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:35.784089    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podStartSLOduration=2.78407463 podStartE2EDuration="2.78407463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:35.781606722 +0000 UTC m=+3365.796168450" watchObservedRunningTime="2026-10-31 04:03:35.78407463 +0000 UTC m=+3365.798636337"
Oct 31 04:03:36.335662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.335639    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:36.342604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.342579    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:36.600852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.600753    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:03:36.601028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.600916    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:03:36.683659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.683631    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:36.684766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:36.684741    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:03:42.520023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.519978    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:03:42.526635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.526609    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.713429 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.659660    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.714388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.713630    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:03:42.714388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.713982    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d8190b\""
Oct 31 04:03:42.714388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.714163    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d8190b-dockercfg-gj2b6\""
Oct 31 04:03:42.714720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.714693    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dt6s\" (UniqueName: \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.815842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.815753    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.816019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.815871    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dt6s\" (UniqueName: \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.816743 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.816720    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.868591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:42.868559    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dt6s\" (UniqueName: \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\") pod \"payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:42.979441 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:42.979408    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:03:43.142866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.142778    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:43.528413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.528374    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:03:43.531262 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:43.531231    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7c0cf7_6685_46ee_b0d9_82b768f91698.slice/crio-076113c4e1b29ae56b44b2941857e4267805eb273f1c65d30ef03f9b6a9f0dff WatchSource:0}: Error finding container 076113c4e1b29ae56b44b2941857e4267805eb273f1c65d30ef03f9b6a9f0dff: Status 404 returned error can't find the container with id 076113c4e1b29ae56b44b2941857e4267805eb273f1c65d30ef03f9b6a9f0dff
Oct 31 04:03:43.744760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.744721    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" event={"ID":"3a7c0cf7-6685-46ee-b0d9-82b768f91698","Type":"ContainerStarted","Data":"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"}
Oct 31 04:03:43.744965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.744812    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" event={"ID":"3a7c0cf7-6685-46ee-b0d9-82b768f91698","Type":"ContainerStarted","Data":"076113c4e1b29ae56b44b2941857e4267805eb273f1c65d30ef03f9b6a9f0dff"}
Oct 31 04:03:43.744965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.744951    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:03:43.851649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.851523    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" podStartSLOduration=1.8515084320000001 podStartE2EDuration="1.851508432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:43.849329432 +0000 UTC m=+3373.863891152" watchObservedRunningTime="2026-10-31 04:03:43.851508432 +0000 UTC m=+3373.866070140"
Oct 31 04:03:43.945363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.945330    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:03:43.949892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:43.949870    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.007597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.000733    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c7585c-openshift-default-dockercfg-jnxt8\""
Oct 31 04:03:44.007597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.005660    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:03:44.033292 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033259    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qfl\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.033663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033632    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.033805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033767    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.033914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033882    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.034050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033958    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.034050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.033986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.034050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.034041    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.034243 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.034065    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.034243 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.034088    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135185    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135234    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135267    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135286    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135318    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135335    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135417    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135521    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qfl\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.135680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.135607    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.136047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.136023    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.136389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.136364    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.136558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.136375    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.136558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.136477    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.136688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.136586    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.138366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.138341    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.138569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.138514    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.188145 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.188110    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.188471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.188444    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qfl\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.264720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.264692    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:44.718494 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:03:44.718453    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3859b3d7_467f_4e5d_a804_85f233dce7d7.slice/crio-f0e80bc39e1938fcc54938b13589a0c925079caadb725b933d696f79e860f140 WatchSource:0}: Error finding container f0e80bc39e1938fcc54938b13589a0c925079caadb725b933d696f79e860f140: Status 404 returned error can't find the container with id f0e80bc39e1938fcc54938b13589a0c925079caadb725b933d696f79e860f140
Oct 31 04:03:44.718494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.718487    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:03:44.721888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.721856    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:44.721990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.721925    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:44.721990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.721956    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:03:44.757454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:44.757421    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" event={"ID":"3859b3d7-467f-4e5d-a804-85f233dce7d7","Type":"ContainerStarted","Data":"f0e80bc39e1938fcc54938b13589a0c925079caadb725b933d696f79e860f140"}
Oct 31 04:03:45.766223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:45.766188    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" event={"ID":"3859b3d7-467f-4e5d-a804-85f233dce7d7","Type":"ContainerStarted","Data":"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"}
Oct 31 04:03:45.967582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:45.967518    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podStartSLOduration=2.967502474 podStartE2EDuration="2.967502474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:45.920605644 +0000 UTC m=+3375.935167355" watchObservedRunningTime="2026-10-31 04:03:45.967502474 +0000 UTC m=+3375.982064182"
Oct 31 04:03:46.265190 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:46.265156    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:46.266920 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:46.266895    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 31 04:03:46.267044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:46.266942    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 31 04:03:46.948742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:46.948706    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" podUID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" containerName="main" containerID="cri-o://360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79" gracePeriod=45
Oct 31 04:03:47.265371 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.265335    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 31 04:03:47.265579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.265400    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 31 04:03:47.324581 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.324548    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 04:03:47.510812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510714    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.510812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510781    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.511078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510820    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\" (UniqueName: \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.511078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510858    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.511078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510897    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.511078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.510984    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\" (UniqueName: \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\") pod \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\" (UID: \"46d92263-e7ce-4fbf-8f15-8fbe7a3de398\") "
Oct 31 04:03:47.511877 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.511831    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:47.512246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.512221    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:47.512367 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.512234    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:47.514098 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.514070    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:47.514508 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.514479    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "kube-api-access-l9d7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:47.514640 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.514566    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm" pod "46d92263-e7ce-4fbf-8f15-8fbe7a3de398" (UID: "46d92263-e7ce-4fbf-8f15-8fbe7a3de398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:03:47.612316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612278    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9d7j\" (UniqueName: \"kubernetes.io/projected/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-kube-api-access-l9d7j\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.612316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612311    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.612316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612322    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.612621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612331    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.612621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612339    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.612621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.612347    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d92263-e7ce-4fbf-8f15-8fbe7a3de398-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:47.814191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.814092    5831 generic.go:356] "Generic (PLEG): container finished" podID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" containerID="360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79" exitCode=0
Oct 31 04:03:47.814191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.814156    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"
Oct 31 04:03:47.814473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.814180    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" event={"ID":"46d92263-e7ce-4fbf-8f15-8fbe7a3de398","Type":"ContainerDied","Data":"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"}
Oct 31 04:03:47.814473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.814230    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj" event={"ID":"46d92263-e7ce-4fbf-8f15-8fbe7a3de398","Type":"ContainerDied","Data":"ad874da37459891366522e6418a970b3e456797cfc309da7aceb9e1171dccf8e"}
Oct 31 04:03:47.814473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.814251    5831 scope.go:122] "RemoveContainer" containerID="360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"
Oct 31 04:03:47.827437 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.827415    5831 scope.go:122] "RemoveContainer" containerID="360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"
Oct 31 04:03:47.827756 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:03:47.827735    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79\": container with ID starting with 360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79 not found: ID does not exist" containerID="360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"
Oct 31 04:03:47.827843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:47.827765    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79"} err="failed to get container status \"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79\": rpc error: code = NotFound desc = could not find container \"360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79\": container with ID starting with 360b98f40d2856b9e0fa17cb5141ab9bdb58c97de470e120bc3a28c68ca4cb79 not found: ID does not exist"
Oct 31 04:03:48.018652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:48.015031    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 04:03:48.033941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:48.025820    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a5f70c/auth-test-model-a5f70c-kserve-5754fc57b9-4x9wj"]
Oct 31 04:03:48.138944 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:48.138863    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" path="/var/lib/kubelet/pods/46d92263-e7ce-4fbf-8f15-8fbe7a3de398/volumes"
Oct 31 04:03:48.265255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:48.265207    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 31 04:03:48.265523 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:48.265495    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 31 04:03:49.269153 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:49.269114    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 31 04:03:49.269663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:49.269189    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 31 04:03:50.265154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:50.265117    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Oct 31 04:03:50.265334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:50.265197    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Oct 31 04:03:51.270026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:51.269998    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:51.270583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:51.270562    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:51.271235 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:51.271206    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:03:54.760710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:03:54.760674    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:04:07.364467 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:07.364425    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:04:12.980834 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:04:12.980789    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:04:28.029269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.029234    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:04:28.030163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.030144    5831 cpu_manager.go:414] "containerMap: removing container" podUID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" containerName="main"
Oct 31 04:04:28.030234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.030166    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" containerName="main"
Oct 31 04:04:28.030284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.030274    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d92263-e7ce-4fbf-8f15-8fbe7a3de398" containerName="main"
Oct 31 04:04:28.034159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.034139    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.117264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.117236    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1f1314-openshift-default-dockercfg-w6bvp\""
Oct 31 04:04:28.136086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136057    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136103    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136139    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136200    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6kn\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136233    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136293    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136362    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136392    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.136517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.136477    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.137389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.137366    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.262908    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263004    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263031    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263114    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263200    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263237    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263271    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263337    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6kn\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263374    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263481    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.270374    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.275619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.271101    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.290579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.263832    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.290579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.283981    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.296552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.291109    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.300636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.300588    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.341215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.341179    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6kn\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.358567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.356087    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\") pod \"e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") " pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:28.648196 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:28.648099    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:29.084167 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.084125    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:04:29.086046 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:04:29.086016    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dfd5a1_079f_40c6_ac0c_e9425090bce5.slice/crio-5e4f1eed584e0543c46aa80fd8898b7cabf7807c5d094b96c25284a0bcb8dd15 WatchSource:0}: Error finding container 5e4f1eed584e0543c46aa80fd8898b7cabf7807c5d094b96c25284a0bcb8dd15: Status 404 returned error can't find the container with id 5e4f1eed584e0543c46aa80fd8898b7cabf7807c5d094b96c25284a0bcb8dd15
Oct 31 04:04:29.088805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.088768    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:04:29.088934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.088853    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:04:29.088934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.088899    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:04:29.197373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.197318    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerStarted","Data":"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"}
Oct 31 04:04:29.197686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.197392    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerStarted","Data":"5e4f1eed584e0543c46aa80fd8898b7cabf7807c5d094b96c25284a0bcb8dd15"}
Oct 31 04:04:29.308650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.308519    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podStartSLOduration=2.308505645 podStartE2EDuration="2.308505645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:29.308010626 +0000 UTC m=+3419.322572334" watchObservedRunningTime="2026-10-31 04:04:29.308505645 +0000 UTC m=+3419.323067344"
Oct 31 04:04:29.649096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:29.649064    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:30.655322 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:30.655292    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:31.216510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:31.216475    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:31.217495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:31.217472    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:04:42.890337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:04:42.890304    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:04:42.980144 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:04:42.980101    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:05:12.680141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.680085    5831 generic.go:356] "Generic (PLEG): container finished" podID="098782ce-3828-401e-890e-a33f83cfca62" containerID="5ad94f85d37db0df07b5ca5dff8ccd17eb62d194db5f2e9f7db905d893d2d617" exitCode=0
Oct 31 04:05:12.680760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.680153    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerDied","Data":"5ad94f85d37db0df07b5ca5dff8ccd17eb62d194db5f2e9f7db905d893d2d617"}
Oct 31 04:05:12.680760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.680705    5831 scope.go:122] "RemoveContainer" containerID="5ad94f85d37db0df07b5ca5dff8ccd17eb62d194db5f2e9f7db905d893d2d617"
Oct 31 04:05:12.682427 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.682397    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:12.682568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.682472    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:12.682568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:12.682502    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:12.990622 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:05:12.990499    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:05:13.719775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:13.719735    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerStarted","Data":"e59b7b32036bd95d8e046a69f9d7355ba924919f4bab166f90d16a8600c0d1bd"}
Oct 31 04:05:14.110685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:14.109909    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:05:14.111094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:14.111062    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 31 04:05:14.111230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:14.111157    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 31 04:05:15.107994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:15.107955    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 31 04:05:15.108672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:15.108627    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 31 04:05:16.107998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:16.107957    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 31 04:05:16.108656 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:16.108623    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 31 04:05:17.112555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:17.109508    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 31 04:05:17.112555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:17.109632    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 31 04:05:18.112577 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:18.111934    5831 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body=
Oct 31 04:05:18.112577 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:18.112001    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused"
Oct 31 04:05:18.507554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:18.506652    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:05:19.114924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:19.114895    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:05:19.116258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:19.116235    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:05:22.566638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:22.566599    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:05:29.640360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.640265    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:05:29.654614 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.653141    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.684929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.684893    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:05:29.734785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.733093    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0d5219\""
Oct 31 04:05:29.734785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.733237    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0d5219-dockercfg-bvf8g\""
Oct 31 04:05:29.757603 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.757561    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5xs\" (UniqueName: \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.757877 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.757853    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.859168 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.859114    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.859388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.859187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5xs\" (UniqueName: \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.860139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.860109    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:29.912793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:29.912719    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5xs\" (UniqueName: \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\") pod \"payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:30.005052 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:30.005013    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:30.388792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:30.388759    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:05:30.390210 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:05:30.390168    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abfb9e6_a260_41df_aad2_05c6cb42299a.slice/crio-d4201e41f2838afffba77c8792975906c57fa88c323457e6ff110c9383da0ea0 WatchSource:0}: Error finding container d4201e41f2838afffba77c8792975906c57fa88c323457e6ff110c9383da0ea0: Status 404 returned error can't find the container with id d4201e41f2838afffba77c8792975906c57fa88c323457e6ff110c9383da0ea0
Oct 31 04:05:30.954583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:30.954542    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" event={"ID":"8abfb9e6-a260-41df-aad2-05c6cb42299a","Type":"ContainerStarted","Data":"995e6427d3728b1edb13297e176df31242d67b03a1f4818ccc33fbe6ba61f89f"}
Oct 31 04:05:30.955134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:30.954605    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" event={"ID":"8abfb9e6-a260-41df-aad2-05c6cb42299a","Type":"ContainerStarted","Data":"d4201e41f2838afffba77c8792975906c57fa88c323457e6ff110c9383da0ea0"}
Oct 31 04:05:30.955134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:30.954664    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:31.062313 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:31.062254    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" podStartSLOduration=2.062240703 podStartE2EDuration="2.062240703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:31.059921836 +0000 UTC m=+3481.074483557" watchObservedRunningTime="2026-10-31 04:05:31.062240703 +0000 UTC m=+3481.076802419"
Oct 31 04:05:33.979736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.979704    5831 generic.go:356] "Generic (PLEG): container finished" podID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerID="6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f" exitCode=0
Oct 31 04:05:33.980192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.979777    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerDied","Data":"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"}
Oct 31 04:05:33.980192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.980184    5831 scope.go:122] "RemoveContainer" containerID="6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"
Oct 31 04:05:33.981922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.981897    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:33.982042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.981964    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:33.982042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:33.981994    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:05:35.018365 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:35.018333    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerStarted","Data":"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"}
Oct 31 04:05:35.335434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:35.335349    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:05:35.336918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:35.336892    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:35.337054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:35.336953    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:36.336195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:36.336160    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:36.336681 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:36.336214    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:36.688694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:36.688660    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:05:37.335780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:37.335736    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:37.336005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:37.335798    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:38.336029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:38.335992    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:38.336475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:38.336049    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:39.335590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:39.335539    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:39.335854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:39.335605    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:40.336406 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:40.336364    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 31 04:05:40.337011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:40.336434    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 31 04:05:41.340607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:41.340578    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:05:41.341671 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:41.341652    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:05:41.965616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:41.965583    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:05:42.979083 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:05:42.979045    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:05:56.426992 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.426955    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:05:56.431792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.431768    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.526222 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.526191    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1f1314-serving-cert\""
Oct 31 04:05:56.530135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.530102    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:05:56.548945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.548911    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.549113 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.549064    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.549113 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.549088    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cth9v\" (UniqueName: \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.549228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.549132    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.650561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.650514    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.650829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.650622    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.650829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.650640    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cth9v\" (UniqueName: \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.650829 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:05:56.650655    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-serving-cert: secret "maas-api-e2e-extmod-b-1f1314-serving-cert" not found
Oct 31 04:05:56.650829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.650678    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.650829 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:05:56.650719    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls podName:152455fe-94b2-43c8-b7dc-2c196657afbc nodeName:}" failed. No retries permitted until 2026-10-31 04:05:57.150701247 +0000 UTC m=+3507.165262932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls") pod "maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" (UID: "152455fe-94b2-43c8-b7dc-2c196657afbc") : secret "maas-api-e2e-extmod-b-1f1314-serving-cert" not found
Oct 31 04:05:56.651481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.651453    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.653355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.653333    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:56.717593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:56.717483    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cth9v\" (UniqueName: \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:57.157178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:57.157087    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:57.159963 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:57.159928    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:57.345883 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:57.345851    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:57.721204 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:57.721171    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:05:57.722940 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:05:57.722908    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152455fe_94b2_43c8_b7dc_2c196657afbc.slice/crio-19705ca900a2f3056472db1f0b6c1379d38faa5323445141ff9d62deea63cd17 WatchSource:0}: Error finding container 19705ca900a2f3056472db1f0b6c1379d38faa5323445141ff9d62deea63cd17: Status 404 returned error can't find the container with id 19705ca900a2f3056472db1f0b6c1379d38faa5323445141ff9d62deea63cd17
Oct 31 04:05:57.724383 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:57.724365    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:05:58.203318 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:58.203236    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" event={"ID":"152455fe-94b2-43c8-b7dc-2c196657afbc","Type":"ContainerStarted","Data":"19705ca900a2f3056472db1f0b6c1379d38faa5323445141ff9d62deea63cd17"}
Oct 31 04:05:59.213140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:59.213095    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" event={"ID":"152455fe-94b2-43c8-b7dc-2c196657afbc","Type":"ContainerStarted","Data":"6e86456cba347d96aff3dc42fca741a1c2aa3e619b8b3445401c56e9ccbfd30a"}
Oct 31 04:05:59.213621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:59.213244    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:05:59.324188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:05:59.324120    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" podStartSLOduration=2.246713913 podStartE2EDuration="3.324103058s" totalImagesPullingTime="1.077389145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:59.319502305 +0000 UTC m=+3509.334064015" watchObservedRunningTime="2026-10-31 04:05:59.324103058 +0000 UTC m=+3509.338664761"
Oct 31 04:06:05.226023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:05.225992    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:06:12.978823 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:06:12.978778    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:06:14.768752 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:14.768715    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:06:25.510935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.510894    5831 generic.go:356] "Generic (PLEG): container finished" podID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerID="48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff" exitCode=0
Oct 31 04:06:25.511489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.510978    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerDied","Data":"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"}
Oct 31 04:06:25.511489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.511482    5831 scope.go:122] "RemoveContainer" containerID="48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"
Oct 31 04:06:25.513162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.513135    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:25.513273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.513201    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:25.513273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:25.513232    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:26.522616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:26.522579    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerStarted","Data":"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"}
Oct 31 04:06:26.648450 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:26.648417    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:06:26.648770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:26.648747    5831 patch_prober.go:29] interesting pod/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 31 04:06:26.648832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:26.648805    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 31 04:06:27.649060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:27.649023    5831 patch_prober.go:29] interesting pod/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 31 04:06:27.649498 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:27.649087    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 31 04:06:28.648497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:28.648457    5831 patch_prober.go:29] interesting pod/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 31 04:06:28.648729 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:28.648563    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 31 04:06:29.649370 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:29.649333    5831 patch_prober.go:29] interesting pod/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 31 04:06:29.649878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:29.649388    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 31 04:06:30.649540 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:30.649486    5831 patch_prober.go:29] interesting pod/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Oct 31 04:06:30.649986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:30.649572    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Oct 31 04:06:31.217295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:31.217261    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:06:31.698480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:31.698448    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:06:31.699660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:31.699638    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:06:36.043129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:36.043088    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"]
Oct 31 04:06:36.095274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:36.095239    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3013f4-bn76n"]
Oct 31 04:06:36.136387 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:36.136357    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba36bab-d797-40d0-b8d7-9f0436fd2e2e" path="/var/lib/kubelet/pods/aba36bab-d797-40d0-b8d7-9f0436fd2e2e/volumes"
Oct 31 04:06:40.046344 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:40.046300    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"]
Oct 31 04:06:40.092392 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:40.092357    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-969f24a4-q7w76"]
Oct 31 04:06:40.136929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:40.136897    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788fde44-7590-4864-8392-0f072d468bd4" path="/var/lib/kubelet/pods/788fde44-7590-4864-8392-0f072d468bd4/volumes"
Oct 31 04:06:42.983841 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:06:42.983784    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:06:44.617644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:44.617599    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:06:45.770250 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.770216    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:06:45.776510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.776489    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.867565 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.864045    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d253a5-openshift-default-dockercfg-b7br4\""
Oct 31 04:06:45.882332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.882289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.911591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.908749    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.940425 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.940306    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.966721    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.966794    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.966854    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.966927    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.966953    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.967022    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:45.976632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:45.967065    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068315    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068400    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068437    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068465    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068505    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068606    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068632    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068689    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.068757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068730    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.070895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068906    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.070895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.068952    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.070895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.069312    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.070895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.069437    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.070895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.069864    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.071575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.071548    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.071697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.071671    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.149613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.140284    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.161552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.155339    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\") pod \"e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") " pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.395609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.394789    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:46.829112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.829082    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:06:46.830445 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:06:46.830419    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeaa8ae1_2b7a_4f47_91e1_e3535ede5bdd.slice/crio-6c60c59e85713639c0e14aa6f5317a1e9dc02aa2907814182fbcf0b157c70ad4 WatchSource:0}: Error finding container 6c60c59e85713639c0e14aa6f5317a1e9dc02aa2907814182fbcf0b157c70ad4: Status 404 returned error can't find the container with id 6c60c59e85713639c0e14aa6f5317a1e9dc02aa2907814182fbcf0b157c70ad4
Oct 31 04:06:46.833238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.833209    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:46.833320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.833272    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:46.833320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:46.833300    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:06:47.712454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:47.712408    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerStarted","Data":"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"}
Oct 31 04:06:47.712680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:47.712475    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerStarted","Data":"6c60c59e85713639c0e14aa6f5317a1e9dc02aa2907814182fbcf0b157c70ad4"}
Oct 31 04:06:47.869704 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:47.869649    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podStartSLOduration=2.869632544 podStartE2EDuration="2.869632544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:47.865471653 +0000 UTC m=+3557.880033361" watchObservedRunningTime="2026-10-31 04:06:47.869632544 +0000 UTC m=+3557.884194251"
Oct 31 04:06:48.396202 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:48.396166    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:48.402587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:48.402561    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:48.723102 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:48.723067    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:48.724163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:48.724138    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:06:49.211655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:49.211622    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 04:06:49.212253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:49.211857    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy" containerID="cri-o://c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd" gracePeriod=30
Oct 31 04:06:49.618049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:49.617924    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" podUID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" containerName="payload-processing" containerID="cri-o://f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a" gracePeriod=25
Oct 31 04:06:49.915068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:49.915037    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:06:50.052795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.052762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\") pod \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") "
Oct 31 04:06:50.052994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.052898    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\" (UniqueName: \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\") pod \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\" (UID: \"3a7c0cf7-6685-46ee-b0d9-82b768f91698\") "
Oct 31 04:06:50.053231 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.053203    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume" pod "3a7c0cf7-6685-46ee-b0d9-82b768f91698" (UID: "3a7c0cf7-6685-46ee-b0d9-82b768f91698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:06:50.053571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.053547    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7c0cf7-6685-46ee-b0d9-82b768f91698-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:50.055311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.055276    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s" pod "3a7c0cf7-6685-46ee-b0d9-82b768f91698" (UID: "3a7c0cf7-6685-46ee-b0d9-82b768f91698"). InnerVolumeSpecName "kube-api-access-6dt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:50.154897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.154867    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dt6s\" (UniqueName: \"kubernetes.io/projected/3a7c0cf7-6685-46ee-b0d9-82b768f91698-kube-api-access-6dt6s\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:50.742260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.742224    5831 generic.go:356] "Generic (PLEG): container finished" podID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" containerID="f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a" exitCode=0
Oct 31 04:06:50.742829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.742289    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"
Oct 31 04:06:50.742829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.742319    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" event={"ID":"3a7c0cf7-6685-46ee-b0d9-82b768f91698","Type":"ContainerDied","Data":"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"}
Oct 31 04:06:50.742829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.742405    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct" event={"ID":"3a7c0cf7-6685-46ee-b0d9-82b768f91698","Type":"ContainerDied","Data":"076113c4e1b29ae56b44b2941857e4267805eb273f1c65d30ef03f9b6a9f0dff"}
Oct 31 04:06:50.742829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.742439    5831 scope.go:122] "RemoveContainer" containerID="f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"
Oct 31 04:06:50.755004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.754970    5831 scope.go:122] "RemoveContainer" containerID="f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"
Oct 31 04:06:50.755299 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:06:50.755276    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a\": container with ID starting with f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a not found: ID does not exist" containerID="f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"
Oct 31 04:06:50.755374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.755308    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a"} err="failed to get container status \"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a\": rpc error: code = NotFound desc = could not find container \"f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a\": container with ID starting with f13d5685b394259c8bf858ca8d903c09d63eb6b716af19cab5f16ce8dec88b3a not found: ID does not exist"
Oct 31 04:06:50.899417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.899379    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:06:50.944384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:50.944350    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8190b-64dfdcf95f-slpct"]
Oct 31 04:06:51.054354 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:51.054267    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"]
Oct 31 04:06:51.060163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:51.060136    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1053b9-khjbd"]
Oct 31 04:06:52.136341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:52.136310    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03184725-b9aa-4be1-be70-c1519ad3b79a" path="/var/lib/kubelet/pods/03184725-b9aa-4be1-be70-c1519ad3b79a/volumes"
Oct 31 04:06:52.136780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:52.136766    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" path="/var/lib/kubelet/pods/3a7c0cf7-6685-46ee-b0d9-82b768f91698/volumes"
Oct 31 04:06:54.662365 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.662334    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 04:06:54.777469 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.777435    5831 generic.go:356] "Generic (PLEG): container finished" podID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerID="c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd" exitCode=0
Oct 31 04:06:54.777669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.777484    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"
Oct 31 04:06:54.777669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.777520    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerDied","Data":"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"}
Oct 31 04:06:54.777669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.777586    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm" event={"ID":"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8","Type":"ContainerDied","Data":"502af64ab73ae3a628b27df4d024ffe53345717ea554e3fbbe765b930586a82f"}
Oct 31 04:06:54.777669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.777610    5831 scope.go:122] "RemoveContainer" containerID="c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"
Oct 31 04:06:54.786921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.786889    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.786938    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787014    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787133    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787262    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787390 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787324    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787390 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787369    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.787505 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787399    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\") pod \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\" (UID: \"67d765aa-a51a-4268-b6a1-5a9f6cc64ee8\") "
Oct 31 04:06:54.788005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787898    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:06:54.788005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787925    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:54.788005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787968    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:54.788005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.787968    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:54.788353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.788047    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:54.789194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.789176    5831 scope.go:122] "RemoveContainer" containerID="4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"
Oct 31 04:06:54.789920 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.789878    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:06:54.790029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.789937    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "kube-api-access-jt6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:54.790029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.789967    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:54.790130 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.790074    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy" pod "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" (UID: "67d765aa-a51a-4268-b6a1-5a9f6cc64ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:54.819515 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.819489    5831 scope.go:122] "RemoveContainer" containerID="c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"
Oct 31 04:06:54.820148 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:06:54.820122    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd\": container with ID starting with c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd not found: ID does not exist" containerID="c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"
Oct 31 04:06:54.820289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.820155    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd"} err="failed to get container status \"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd\": rpc error: code = NotFound desc = could not find container \"c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd\": container with ID starting with c5e2f3f10299c0f776e4d326bacccdead8e29399d0de4a2d04a64227e2d802fd not found: ID does not exist"
Oct 31 04:06:54.820289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.820186    5831 scope.go:122] "RemoveContainer" containerID="4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"
Oct 31 04:06:54.821163 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:06:54.820873    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3\": container with ID starting with 4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3 not found: ID does not exist" containerID="4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"
Oct 31 04:06:54.821163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.820908    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3"} err="failed to get container status \"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3\": rpc error: code = NotFound desc = could not find container \"4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3\": container with ID starting with 4e74fd318d41fd1b41f32dcfb9a43b38f1763038b18a1f75c062a9d27093dbd3 not found: ID does not exist"
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888897    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888924    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888934    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888942    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888951    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888959    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt6xg\" (UniqueName: \"kubernetes.io/projected/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-kube-api-access-jt6xg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888968    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888976    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:54.888984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:54.888984    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:55.208643 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:55.208610    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 04:06:55.212673 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:55.212647    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-9m5qm"]
Oct 31 04:06:56.136609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:06:56.136574    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" path="/var/lib/kubelet/pods/67d765aa-a51a-4268-b6a1-5a9f6cc64ee8/volumes"
Oct 31 04:07:12.978987 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:07:12.978944    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:07:18.348105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:18.348070    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:07:36.392159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:36.392075    5831 scope.go:122] "RemoveContainer" containerID="b116961beb5c036fdd9ec08b561f14dd1b37aaf1f05227724a7e79fe1c732910"
Oct 31 04:07:38.049913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.049875    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"]
Oct 31 04:07:38.089310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.089274    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d734a583-fwv5x"]
Oct 31 04:07:38.147908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.147877    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e083eb13-c637-46b0-9dc1-f9aab5bbdd72" path="/var/lib/kubelet/pods/e083eb13-c637-46b0-9dc1-f9aab5bbdd72/volumes"
Oct 31 04:07:38.485236 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.485210    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:07:38.490890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.490864    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:07:38.494893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.494869    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:07:38.498738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.498717    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:07:38.500191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.500169    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:07:38.503179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:07:38.503161    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:07:42.980745 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:07:42.980703    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:08:10.069894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:10.069857    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:08:10.070448 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:10.070109    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" containerID="cri-o://ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a" gracePeriod=30
Oct 31 04:08:11.343706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:11.343666    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:08:11.344104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:11.343740    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:08:11.547870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:11.547830    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:08:12.981361 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:12.981326    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:08:15.378865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.378837    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:08:15.470928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.470889    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.470938    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.470981    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471030    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471113    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471168    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471224    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471244    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471283    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471300    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:15.471401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471359    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\") pod \"9dfd2d19-2fb5-4463-858a-36831814efb8\" (UID: \"9dfd2d19-2fb5-4463-858a-36831814efb8\") "
Oct 31 04:08:15.471780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471605    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:15.471780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471613    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:15.471898 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.471871    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:15.472246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.472035    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.472246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.472058    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.472246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.472072    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.472246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.472085    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd2d19-2fb5-4463-858a-36831814efb8-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.472246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.472098    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.473806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.473783    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:08:15.474133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.474117    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "kube-api-access-sm2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:15.474302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.474282    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:15.474502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.474485    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token" pod "9dfd2d19-2fb5-4463-858a-36831814efb8" (UID: "9dfd2d19-2fb5-4463-858a-36831814efb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:15.503956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.503918    5831 generic.go:356] "Generic (PLEG): container finished" podID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerID="ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a" exitCode=0
Oct 31 04:08:15.504093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.503968    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"
Oct 31 04:08:15.504093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.503987    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerDied","Data":"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"}
Oct 31 04:08:15.504093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.504046    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh" event={"ID":"9dfd2d19-2fb5-4463-858a-36831814efb8","Type":"ContainerDied","Data":"0d5f10cee9495e656f284be565592b4af33ce5dbd45377b87a2756cfaa71d56a"}
Oct 31 04:08:15.504093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.504078    5831 scope.go:122] "RemoveContainer" containerID="ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"
Oct 31 04:08:15.515677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.515660    5831 scope.go:122] "RemoveContainer" containerID="6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"
Oct 31 04:08:15.526060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.526043    5831 scope.go:122] "RemoveContainer" containerID="ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"
Oct 31 04:08:15.526326 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:15.526305    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a\": container with ID starting with ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a not found: ID does not exist" containerID="ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"
Oct 31 04:08:15.526412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.526337    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a"} err="failed to get container status \"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a\": rpc error: code = NotFound desc = could not find container \"ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a\": container with ID starting with ac6f532d80e318fba7e215a654c215e5aad9e5314c51aceefe3afa500a9a105a not found: ID does not exist"
Oct 31 04:08:15.526412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.526361    5831 scope.go:122] "RemoveContainer" containerID="6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"
Oct 31 04:08:15.526626 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:15.526602    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f\": container with ID starting with 6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f not found: ID does not exist" containerID="6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"
Oct 31 04:08:15.526683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.526631    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f"} err="failed to get container status \"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f\": rpc error: code = NotFound desc = could not find container \"6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f\": container with ID starting with 6210ad761d51dd6f9a0a3af2a8accf8f8dfa6b3009a89193a5ca18f1b2d8e96f not found: ID does not exist"
Oct 31 04:08:15.583413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.578716    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.583413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.578779    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.583413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.578799    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.583413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.578812    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2l8\" (UniqueName: \"kubernetes.io/projected/9dfd2d19-2fb5-4463-858a-36831814efb8-kube-api-access-sm2l8\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:15.639348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.639262    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:08:15.645586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:15.645561    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8190b-openshift-default-b6499c977-lf4mh"]
Oct 31 04:08:16.136914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:16.136873    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" path="/var/lib/kubelet/pods/9dfd2d19-2fb5-4463-858a-36831814efb8/volumes"
Oct 31 04:08:20.147367 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.147338    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:08:20.148198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148176    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:08:20.148314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148200    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:08:20.148314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148224    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" containerName="payload-processing"
Oct 31 04:08:20.148314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148232    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" containerName="payload-processing"
Oct 31 04:08:20.148314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148243    5831 cpu_manager.go:414] "containerMap: removing container" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:08:20.148314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148250    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:08:20.148630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148479    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:08:20.148630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148506    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:08:20.148630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148522    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:08:20.148630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.148559    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7c0cf7-6685-46ee-b0d9-82b768f91698" containerName="payload-processing"
Oct 31 04:08:20.153773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.153749    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.201711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.201685    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d253a5-serving-cert\""
Oct 31 04:08:20.250332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.250299    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tp4\" (UniqueName: \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.250524 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.250379    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.250635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.250603    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.250761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.250739    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.251626 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.251595    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:08:20.352561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.352490    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.352759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.352622    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.352759 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:20.352661    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-serving-cert: secret "maas-api-e2e-extmod-a-d253a5-serving-cert" not found
Oct 31 04:08:20.352759 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:20.352730    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls podName:49e77eea-bf94-4685-aa6c-355f4910356b nodeName:}" failed. No retries permitted until 2026-10-31 04:08:20.852712629 +0000 UTC m=+3650.867274335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls") pod "maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" (UID: "49e77eea-bf94-4685-aa6c-355f4910356b") : secret "maas-api-e2e-extmod-a-d253a5-serving-cert" not found
Oct 31 04:08:20.352960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.352775    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tp4\" (UniqueName: \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.352960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.352851    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.353567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.353546    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.355337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.355320    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.432554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.431358    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tp4\" (UniqueName: \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.860500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.860413    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:20.863144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:20.863117    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:21.069443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:21.069407    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:21.453010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:21.452914    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:08:21.558585 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:21.558521    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" event={"ID":"49e77eea-bf94-4685-aa6c-355f4910356b","Type":"ContainerStarted","Data":"f9d2262e800263a6032dfc59663988156093e60e3c2943e3c14af9b7d51ab950"}
Oct 31 04:08:22.568522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:22.568483    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" event={"ID":"49e77eea-bf94-4685-aa6c-355f4910356b","Type":"ContainerStarted","Data":"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"}
Oct 31 04:08:22.568974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:22.568588    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:22.685647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:22.685581    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" podStartSLOduration=1.739542567 podStartE2EDuration="2.685562123s" totalImagesPullingTime="946.019556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:22.684181752 +0000 UTC m=+3652.698743476" watchObservedRunningTime="2026-10-31 04:08:22.685562123 +0000 UTC m=+3652.700123833"
Oct 31 04:08:28.583336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:28.583298    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:08:30.137141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:30.137109    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:08:30.245039 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:30.244988    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:08:35.137908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.137865    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" podUID="34eaa693-87be-4f7b-989d-c504b0e14ad1" containerName="payload-processing" containerID="cri-o://e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249" gracePeriod=25
Oct 31 04:08:35.246317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.246275    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" podUID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" containerName="payload-pre-processing" containerID="cri-o://b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3" gracePeriod=25
Oct 31 04:08:35.430699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.430668    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:08:35.485108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.485075    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\" (UniqueName: \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\") pod \"34eaa693-87be-4f7b-989d-c504b0e14ad1\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") "
Oct 31 04:08:35.485290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.485266    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\") pod \"34eaa693-87be-4f7b-989d-c504b0e14ad1\" (UID: \"34eaa693-87be-4f7b-989d-c504b0e14ad1\") "
Oct 31 04:08:35.485799 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.485761    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume" pod "34eaa693-87be-4f7b-989d-c504b0e14ad1" (UID: "34eaa693-87be-4f7b-989d-c504b0e14ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:35.485958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.485930    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34eaa693-87be-4f7b-989d-c504b0e14ad1-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:35.489688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.489661    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z" pod "34eaa693-87be-4f7b-989d-c504b0e14ad1" (UID: "34eaa693-87be-4f7b-989d-c504b0e14ad1"). InnerVolumeSpecName "kube-api-access-tqr6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:35.525398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.525376    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:08:35.587035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.586996    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\" (UniqueName: \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\") pod \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") "
Oct 31 04:08:35.587235 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.587123    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\") pod \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\" (UID: \"4ad987cc-6640-4085-85c7-8d1d2c2c3707\") "
Oct 31 04:08:35.587482 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.587460    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume" pod "4ad987cc-6640-4085-85c7-8d1d2c2c3707" (UID: "4ad987cc-6640-4085-85c7-8d1d2c2c3707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:35.587632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.587614    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad987cc-6640-4085-85c7-8d1d2c2c3707-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:35.587702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.587639    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqr6z\" (UniqueName: \"kubernetes.io/projected/34eaa693-87be-4f7b-989d-c504b0e14ad1-kube-api-access-tqr6z\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:35.589308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.589288    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6" pod "4ad987cc-6640-4085-85c7-8d1d2c2c3707" (UID: "4ad987cc-6640-4085-85c7-8d1d2c2c3707"). InnerVolumeSpecName "kube-api-access-xtkq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:35.677674 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.677582    5831 generic.go:356] "Generic (PLEG): container finished" podID="34eaa693-87be-4f7b-989d-c504b0e14ad1" containerID="e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249" exitCode=0
Oct 31 04:08:35.677882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.677679    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"
Oct 31 04:08:35.677882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.677757    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" event={"ID":"34eaa693-87be-4f7b-989d-c504b0e14ad1","Type":"ContainerDied","Data":"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"}
Oct 31 04:08:35.677882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.677817    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62" event={"ID":"34eaa693-87be-4f7b-989d-c504b0e14ad1","Type":"ContainerDied","Data":"412aa49d2c0b89a923310bce0cec73ab9b11fce25e167ff70a56dfe77ebaed7d"}
Oct 31 04:08:35.677882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.677850    5831 scope.go:122] "RemoveContainer" containerID="e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"
Oct 31 04:08:35.679562 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.679499    5831 generic.go:356] "Generic (PLEG): container finished" podID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" containerID="b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3" exitCode=0
Oct 31 04:08:35.679695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.679574    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"
Oct 31 04:08:35.679695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.679575    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" event={"ID":"4ad987cc-6640-4085-85c7-8d1d2c2c3707","Type":"ContainerDied","Data":"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"}
Oct 31 04:08:35.679799 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.679703    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk" event={"ID":"4ad987cc-6640-4085-85c7-8d1d2c2c3707","Type":"ContainerDied","Data":"98a58c274a5061f11f77c8b63f58b15b9964897fe2e3feb1674900bd026d106b"}
Oct 31 04:08:35.688456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.688431    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkq6\" (UniqueName: \"kubernetes.io/projected/4ad987cc-6640-4085-85c7-8d1d2c2c3707-kube-api-access-xtkq6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:35.690998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.690420    5831 scope.go:122] "RemoveContainer" containerID="e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"
Oct 31 04:08:35.690998 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:35.690810    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249\": container with ID starting with e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249 not found: ID does not exist" containerID="e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"
Oct 31 04:08:35.690998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.690848    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249"} err="failed to get container status \"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249\": rpc error: code = NotFound desc = could not find container \"e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249\": container with ID starting with e036accfe430f14ea72da03ca5ee31716ef9d1df21a4a5ebc0c21ca21ea7b249 not found: ID does not exist"
Oct 31 04:08:35.690998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.690873    5831 scope.go:122] "RemoveContainer" containerID="b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"
Oct 31 04:08:35.701336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.701315    5831 scope.go:122] "RemoveContainer" containerID="b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"
Oct 31 04:08:35.701633 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:35.701602    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3\": container with ID starting with b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3 not found: ID does not exist" containerID="b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"
Oct 31 04:08:35.701713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.701634    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3"} err="failed to get container status \"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3\": rpc error: code = NotFound desc = could not find container \"b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3\": container with ID starting with b1847c640396a385cf520efe4b508c5d35df1ff3e92db7181e81cf209e9344b3 not found: ID does not exist"
Oct 31 04:08:35.868146 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.868107    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:08:35.871934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.871909    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8fa21-7874d5664c-6lxkk"]
Oct 31 04:08:35.972290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.972259    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:08:35.975723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:35.975692    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8fa21-7c8ddb7cd9-82k62"]
Oct 31 04:08:36.136841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.136809    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34eaa693-87be-4f7b-989d-c504b0e14ad1" path="/var/lib/kubelet/pods/34eaa693-87be-4f7b-989d-c504b0e14ad1/volumes"
Oct 31 04:08:36.137347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.137326    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" path="/var/lib/kubelet/pods/4ad987cc-6640-4085-85c7-8d1d2c2c3707/volumes"
Oct 31 04:08:36.437596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.437561    5831 scope.go:122] "RemoveContainer" containerID="271abe3623dab91b49a301a748ae060bbb8c0bfe25130c56e00eac93cb20f271"
Oct 31 04:08:36.448607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.448585    5831 scope.go:122] "RemoveContainer" containerID="8615c03cb2418224670c7854a217e5f2164bc377a5e351f88030c2aec8591ae2"
Oct 31 04:08:36.458347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.458327    5831 scope.go:122] "RemoveContainer" containerID="846a09ba21f66c3ac3675a30745a0b77bff75b3affa3f9f076696d61eea4ed97"
Oct 31 04:08:36.467873 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:36.467854    5831 scope.go:122] "RemoveContainer" containerID="80be0a736687bd06d15a390acc20ba731003deaf769c936c87a0fe1ad2c3cf8d"
Oct 31 04:08:42.980572 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:08:42.980499    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:08:45.386066 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:45.386018    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:08:57.008001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.007956    5831 generic.go:356] "Generic (PLEG): container finished" podID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerID="5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f" exitCode=0
Oct 31 04:08:57.008621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.008077    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerDied","Data":"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"}
Oct 31 04:08:57.008621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.008602    5831 scope.go:122] "RemoveContainer" containerID="5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"
Oct 31 04:08:57.010646 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.010614    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:08:57.010781 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.010692    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:08:57.010781 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:57.010726    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:08:58.018345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:58.018307    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerStarted","Data":"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"}
Oct 31 04:08:58.403800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:58.400699    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:08:58.403800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:58.400841    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 31 04:08:58.403800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:58.400885    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 31 04:08:59.400555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:59.399932    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 31 04:08:59.400555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:08:59.400002    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 31 04:09:00.209666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:00.209629    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:09:00.209929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:00.209905    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy" containerID="cri-o://18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166" gracePeriod=30
Oct 31 04:09:00.398554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:00.397964    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 31 04:09:00.398554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:00.398025    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 31 04:09:01.402555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:01.401991    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 31 04:09:01.402555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:01.402056    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 31 04:09:02.402553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:02.401929    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body=
Oct 31 04:09:02.402553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:02.401995    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused"
Oct 31 04:09:03.420187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:03.420152    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:09:03.420828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:03.420803    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:09:03.422006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:03.421983    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:09:05.496938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.496871    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:09:05.619311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619278    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619333    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619382    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619441    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619480    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619609    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619652    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619670    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.619811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619703    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.619811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619779    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\") pod \"3859b3d7-467f-4e5d-a804-85f233dce7d7\" (UID: \"3859b3d7-467f-4e5d-a804-85f233dce7d7\") "
Oct 31 04:09:05.620084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.619849    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.620280 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.620257    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.620463 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.620446    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.620587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.620447    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:05.620913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.620883    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.621004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.620958    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.628015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.627988    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:09:05.628015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.627999    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:05.628361 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.628059    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:09:05.630114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.630091    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl" pod "3859b3d7-467f-4e5d-a804-85f233dce7d7" (UID: "3859b3d7-467f-4e5d-a804-85f233dce7d7"). InnerVolumeSpecName "kube-api-access-94qfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:05.721887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721845    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qfl\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-kube-api-access-94qfl\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.721887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721882    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3859b3d7-467f-4e5d-a804-85f233dce7d7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.721887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721895    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.722155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721908    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.722155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721919    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.722155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721930    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:05.722155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:05.721941    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3859b3d7-467f-4e5d-a804-85f233dce7d7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:06.091868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.091829    5831 generic.go:356] "Generic (PLEG): container finished" podID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerID="18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166" exitCode=0
Oct 31 04:09:06.092084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.091890    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"
Oct 31 04:09:06.092084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.091911    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" event={"ID":"3859b3d7-467f-4e5d-a804-85f233dce7d7","Type":"ContainerDied","Data":"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"}
Oct 31 04:09:06.092084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.091967    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx" event={"ID":"3859b3d7-467f-4e5d-a804-85f233dce7d7","Type":"ContainerDied","Data":"f0e80bc39e1938fcc54938b13589a0c925079caadb725b933d696f79e860f140"}
Oct 31 04:09:06.092084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.091989    5831 scope.go:122] "RemoveContainer" containerID="18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"
Oct 31 04:09:06.106921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.106894    5831 scope.go:122] "RemoveContainer" containerID="18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"
Oct 31 04:09:06.107289 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:09:06.107269    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166\": container with ID starting with 18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166 not found: ID does not exist" containerID="18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"
Oct 31 04:09:06.107342 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.107299    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166"} err="failed to get container status \"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166\": rpc error: code = NotFound desc = could not find container \"18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166\": container with ID starting with 18466204a7368832795ff598917fbc3ae538c71a407a3b67221312560a85b166 not found: ID does not exist"
Oct 31 04:09:06.252922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.252168    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:09:06.295208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:06.295169    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-kfbdx"]
Oct 31 04:09:08.159233 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:08.158430    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" path="/var/lib/kubelet/pods/3859b3d7-467f-4e5d-a804-85f233dce7d7/volumes"
Oct 31 04:09:13.025313 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:09:13.025265    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:09:25.388245 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:25.388201    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:09:25.516309 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:25.516274    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:09:28.106968 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.106928    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:09:28.108185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108150    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:09:28.108185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108183    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfd2d19-2fb5-4463-858a-36831814efb8" containerName="istio-proxy"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108238    5831 cpu_manager.go:414] "containerMap: removing container" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108247    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108256    5831 cpu_manager.go:414] "containerMap: removing container" podUID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" containerName="payload-pre-processing"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108264    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" containerName="payload-pre-processing"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108340    5831 cpu_manager.go:414] "containerMap: removing container" podUID="34eaa693-87be-4f7b-989d-c504b0e14ad1" containerName="payload-processing"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108348    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eaa693-87be-4f7b-989d-c504b0e14ad1" containerName="payload-processing"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108366    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy"
Oct 31 04:09:28.108407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108375    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy"
Oct 31 04:09:28.108945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.108922    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad987cc-6640-4085-85c7-8d1d2c2c3707" containerName="payload-pre-processing"
Oct 31 04:09:28.109140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.109127    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eaa693-87be-4f7b-989d-c504b0e14ad1" containerName="payload-processing"
Oct 31 04:09:28.109230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.109220    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3859b3d7-467f-4e5d-a804-85f233dce7d7" containerName="istio-proxy"
Oct 31 04:09:28.109321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.109312    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d765aa-a51a-4268-b6a1-5a9f6cc64ee8" containerName="istio-proxy"
Oct 31 04:09:28.115215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.115192    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198372    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198415    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198559    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198599    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198705    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198821 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198761    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnc6c\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198873    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.198962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198925    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.199004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.198975    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300095 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300054    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6c\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300116    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300158    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300185    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300265    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300287    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300626 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300346    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300626 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300395    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300626 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300483    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300618    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300776    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.300938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300898    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.301010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.300960    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.301430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.301406    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.303121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.303092    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.303610 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.303583    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.307636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.307600    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:09:28.352743 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.352707    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.352930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.352914    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnc6c\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\") pod \"maas-default-gateway-openshift-default-6886b96f9-fwqzv\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.438723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.438693    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:28.862278 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.862254    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:09:28.864272 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:09:28.864232    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92eedbd_2b47_4315_8128_4c2ddc883d00.slice/crio-d7bb6589051df5aa539c1d9184df9ce5652f5f6c1af0cd5c8d2c2946dc9c746d WatchSource:0}: Error finding container d7bb6589051df5aa539c1d9184df9ce5652f5f6c1af0cd5c8d2c2946dc9c746d: Status 404 returned error can't find the container with id d7bb6589051df5aa539c1d9184df9ce5652f5f6c1af0cd5c8d2c2946dc9c746d
Oct 31 04:09:28.867114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.867077    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:28.867228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.867159    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:28.867228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:28.867206    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:29.360707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.359879    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" event={"ID":"d92eedbd-2b47-4315-8128-4c2ddc883d00","Type":"ContainerStarted","Data":"1e08e898d844b7ef8cebd16cd3cd3b39c974757c9841a8e9cccce1958262a28c"}
Oct 31 04:09:29.360707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.359933    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" event={"ID":"d92eedbd-2b47-4315-8128-4c2ddc883d00","Type":"ContainerStarted","Data":"d7bb6589051df5aa539c1d9184df9ce5652f5f6c1af0cd5c8d2c2946dc9c746d"}
Oct 31 04:09:29.439629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.439587    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:29.441416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.441386    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:29.441567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.441443    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:29.540306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:29.540256    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podStartSLOduration=1.540242187 podStartE2EDuration="1.540242187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:29.512491137 +0000 UTC m=+3719.527052875" watchObservedRunningTime="2026-10-31 04:09:29.540242187 +0000 UTC m=+3719.554803894"
Oct 31 04:09:30.398692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.398605    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" podUID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" containerName="payload-processing" containerID="cri-o://08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938" gracePeriod=25
Oct 31 04:09:30.439955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.439926    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:30.440109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.439979    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:30.516795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.516754    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" podUID="5e44c584-a865-408b-a149-fef2ad92cdac" containerName="payload-pre-processing" containerID="cri-o://b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce" gracePeriod=25
Oct 31 04:09:30.753070 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.753044    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:09:30.854982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.854948    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\") pod \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") "
Oct 31 04:09:30.855162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.855103    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\" (UniqueName: \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\") pod \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\" (UID: \"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e\") "
Oct 31 04:09:30.855442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.855413    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume" pod "9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" (UID: "9dbe0bca-581a-4bc4-add3-eb5c64b7f95e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:30.855601 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.855581    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:30.857689 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.857659    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb" pod "9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" (UID: "9dbe0bca-581a-4bc4-add3-eb5c64b7f95e"). InnerVolumeSpecName "kube-api-access-rx5nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:30.907191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.907154    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:09:30.956618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:30.956582    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5nb\" (UniqueName: \"kubernetes.io/projected/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e-kube-api-access-rx5nb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:31.057203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.057134    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\" (UniqueName: \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\") pod \"5e44c584-a865-408b-a149-fef2ad92cdac\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") "
Oct 31 04:09:31.057389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.057215    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\") pod \"5e44c584-a865-408b-a149-fef2ad92cdac\" (UID: \"5e44c584-a865-408b-a149-fef2ad92cdac\") "
Oct 31 04:09:31.057631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.057596    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume" pod "5e44c584-a865-408b-a149-fef2ad92cdac" (UID: "5e44c584-a865-408b-a149-fef2ad92cdac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:31.057749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.057725    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44c584-a865-408b-a149-fef2ad92cdac-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:31.059680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.059650    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9" pod "5e44c584-a865-408b-a149-fef2ad92cdac" (UID: "5e44c584-a865-408b-a149-fef2ad92cdac"). InnerVolumeSpecName "kube-api-access-2wqm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:31.159068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.159028    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqm9\" (UniqueName: \"kubernetes.io/projected/5e44c584-a865-408b-a149-fef2ad92cdac-kube-api-access-2wqm9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:31.376958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.376856    5831 generic.go:356] "Generic (PLEG): container finished" podID="5e44c584-a865-408b-a149-fef2ad92cdac" containerID="b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce" exitCode=0
Oct 31 04:09:31.376958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.376924    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"
Oct 31 04:09:31.377209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.376950    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" event={"ID":"5e44c584-a865-408b-a149-fef2ad92cdac","Type":"ContainerDied","Data":"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"}
Oct 31 04:09:31.377209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.377032    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp" event={"ID":"5e44c584-a865-408b-a149-fef2ad92cdac","Type":"ContainerDied","Data":"ec88941846d59952ec4c9e290d10ac1a05f1278930bfc5c6f73b763ac8cffbd0"}
Oct 31 04:09:31.377209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.377049    5831 scope.go:122] "RemoveContainer" containerID="b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"
Oct 31 04:09:31.401553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.378621    5831 generic.go:356] "Generic (PLEG): container finished" podID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" containerID="08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938" exitCode=0
Oct 31 04:09:31.401553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.379574    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"
Oct 31 04:09:31.402323 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.402286    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" event={"ID":"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e","Type":"ContainerDied","Data":"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"}
Oct 31 04:09:31.402694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.402668    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8" event={"ID":"9dbe0bca-581a-4bc4-add3-eb5c64b7f95e","Type":"ContainerDied","Data":"9d258ed6b5fd724af50c42eabfec37612110e2615a7c7384ec1f4ba2af566e20"}
Oct 31 04:09:31.413766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.413510    5831 scope.go:122] "RemoveContainer" containerID="b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"
Oct 31 04:09:31.413933 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:09:31.413903    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce\": container with ID starting with b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce not found: ID does not exist" containerID="b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"
Oct 31 04:09:31.414009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.413947    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce"} err="failed to get container status \"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce\": rpc error: code = NotFound desc = could not find container \"b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce\": container with ID starting with b07f653cc5469284f94753f3ec849d8efea1eed016f41624d63fabce22e191ce not found: ID does not exist"
Oct 31 04:09:31.414009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.413973    5831 scope.go:122] "RemoveContainer" containerID="08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"
Oct 31 04:09:31.434679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.434659    5831 scope.go:122] "RemoveContainer" containerID="08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"
Oct 31 04:09:31.434981 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:09:31.434962    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938\": container with ID starting with 08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938 not found: ID does not exist" containerID="08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"
Oct 31 04:09:31.435039 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.434989    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938"} err="failed to get container status \"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938\": rpc error: code = NotFound desc = could not find container \"08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938\": container with ID starting with 08bcdae19cafd101ec99b99735d6fea478f79fb1541dfb4501d07bc6f6c53938 not found: ID does not exist"
Oct 31 04:09:31.439331 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.439310    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:31.439416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.439354    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:31.578137 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.578105    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:09:31.604685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.604644    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5901fe-769bf9697c-c4dr8"]
Oct 31 04:09:31.699555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.692592    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:09:31.709556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:31.709410    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5901fe-7cffc6c69f-4xztp"]
Oct 31 04:09:32.136519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:32.136440    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e44c584-a865-408b-a149-fef2ad92cdac" path="/var/lib/kubelet/pods/5e44c584-a865-408b-a149-fef2ad92cdac/volumes"
Oct 31 04:09:32.136905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:32.136892    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" path="/var/lib/kubelet/pods/9dbe0bca-581a-4bc4-add3-eb5c64b7f95e/volumes"
Oct 31 04:09:32.439757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:32.439725    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:32.440165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:32.439787    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:33.441125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:33.439744    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:33.441125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:33.439804    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:34.471732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:34.471000    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:34.471732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:34.471106    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:35.439169 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:35.439128    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:35.439345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:35.439216    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:36.439669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:36.439625    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Oct 31 04:09:36.440210 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:36.439699    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Oct 31 04:09:38.439859 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:38.439804    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:38.440272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:38.439889    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:39.480703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:39.480659    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:39.481306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:39.480714    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:40.075637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:40.075599    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:40.076067 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:40.076044    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:40.077063 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:40.077043    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:09:42.978477 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:09:42.978435    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:09:45.157481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.157442    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:09:45.158306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158284    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5e44c584-a865-408b-a149-fef2ad92cdac" containerName="payload-pre-processing"
Oct 31 04:09:45.158306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158306    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e44c584-a865-408b-a149-fef2ad92cdac" containerName="payload-pre-processing"
Oct 31 04:09:45.158453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158319    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" containerName="payload-processing"
Oct 31 04:09:45.158453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158325    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" containerName="payload-processing"
Oct 31 04:09:45.158453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158444    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e44c584-a865-408b-a149-fef2ad92cdac" containerName="payload-pre-processing"
Oct 31 04:09:45.158453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.158452    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbe0bca-581a-4bc4-add3-eb5c64b7f95e" containerName="payload-processing"
Oct 31 04:09:45.163678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.163660    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.243655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243613    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.243868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243669    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.243868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243762    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.243868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243847    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243937    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhzv\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.243986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.244027    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.244068    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.244106    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.244564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.244517    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:09:45.278787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.278757    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f0e007-openshift-default-dockercfg-vlc2j\""
Oct 31 04:09:45.357569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357332    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhzv\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.357569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357410    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.357569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357443    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.357569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357483    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.357569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357555    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.358042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357663    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.358042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357692    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.358042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357755    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.358042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.357825    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.367578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.358301    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.367578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.359089    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.367578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.360122    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.367578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.360367    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.376579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.368414    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.376579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.368725    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.395555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.392323    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.448554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.435146    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.448554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.436275    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhzv\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\") pod \"e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") " pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.480173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.480138    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:45.958571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.957796    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:09:45.965943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.965896    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:45.966090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.965981    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:45.966090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:45.966011    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:09:46.555312 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:46.555280    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerStarted","Data":"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"}
Oct 31 04:09:46.555780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:46.555334    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerStarted","Data":"9399fbc53eb5939ac2230ef873aa491721f34ea59519e49e148dd4e2736d10fd"}
Oct 31 04:09:46.666633 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:46.666580    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podStartSLOduration=1.666567033 podStartE2EDuration="1.666567033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:46.663370393 +0000 UTC m=+3736.677932101" watchObservedRunningTime="2026-10-31 04:09:46.666567033 +0000 UTC m=+3736.681128740"
Oct 31 04:09:47.481353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:47.481320    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:47.488228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:47.488196    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:47.564607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:47.564561    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:47.565497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:47.565474    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:09:47.600127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:09:47.600090    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:10:02.606368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:02.606330    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:10:07.606840 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.606795    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" podUID="8abfb9e6-a260-41df-aad2-05c6cb42299a" containerName="payload-processing" containerID="cri-o://995e6427d3728b1edb13297e176df31242d67b03a1f4818ccc33fbe6ba61f89f" gracePeriod=25
Oct 31 04:10:07.832043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.832009    5831 generic.go:356] "Generic (PLEG): container finished" podID="8abfb9e6-a260-41df-aad2-05c6cb42299a" containerID="995e6427d3728b1edb13297e176df31242d67b03a1f4818ccc33fbe6ba61f89f" exitCode=0
Oct 31 04:10:07.832224 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.832088    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" event={"ID":"8abfb9e6-a260-41df-aad2-05c6cb42299a","Type":"ContainerDied","Data":"995e6427d3728b1edb13297e176df31242d67b03a1f4818ccc33fbe6ba61f89f"}
Oct 31 04:10:07.897218 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.897182    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:10:07.995219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.995186    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\") pod \"8abfb9e6-a260-41df-aad2-05c6cb42299a\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") "
Oct 31 04:10:07.995414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.995280    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\" (UniqueName: \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\") pod \"8abfb9e6-a260-41df-aad2-05c6cb42299a\" (UID: \"8abfb9e6-a260-41df-aad2-05c6cb42299a\") "
Oct 31 04:10:07.995721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.995687    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume" pod "8abfb9e6-a260-41df-aad2-05c6cb42299a" (UID: "8abfb9e6-a260-41df-aad2-05c6cb42299a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:07.997657 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:07.997626    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs" pod "8abfb9e6-a260-41df-aad2-05c6cb42299a" (UID: "8abfb9e6-a260-41df-aad2-05c6cb42299a"). InnerVolumeSpecName "kube-api-access-5s5xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:08.096979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:08.096946    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abfb9e6-a260-41df-aad2-05c6cb42299a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:08.096979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:08.096975    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5xs\" (UniqueName: \"kubernetes.io/projected/8abfb9e6-a260-41df-aad2-05c6cb42299a-kube-api-access-5s5xs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:08.840694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:08.840658    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"
Oct 31 04:10:08.841135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:08.840693    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t" event={"ID":"8abfb9e6-a260-41df-aad2-05c6cb42299a","Type":"ContainerDied","Data":"d4201e41f2838afffba77c8792975906c57fa88c323457e6ff110c9383da0ea0"}
Oct 31 04:10:08.841135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:08.840760    5831 scope.go:122] "RemoveContainer" containerID="995e6427d3728b1edb13297e176df31242d67b03a1f4818ccc33fbe6ba61f89f"
Oct 31 04:10:09.000245 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:09.000208    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:10:09.045432 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:09.045399    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d5219-767499cc67-vpv4t"]
Oct 31 04:10:10.137105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:10.137068    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abfb9e6-a260-41df-aad2-05c6cb42299a" path="/var/lib/kubelet/pods/8abfb9e6-a260-41df-aad2-05c6cb42299a/volumes"
Oct 31 04:10:12.979375 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:10:12.979338    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:10:29.724448 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:29.724336    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:10:29.724961 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:29.724684    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" podUID="152455fe-94b2-43c8-b7dc-2c196657afbc" containerName="maas-api" containerID="cri-o://6e86456cba347d96aff3dc42fca741a1c2aa3e619b8b3445401c56e9ccbfd30a" gracePeriod=30
Oct 31 04:10:30.040977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.040892    5831 generic.go:356] "Generic (PLEG): container finished" podID="152455fe-94b2-43c8-b7dc-2c196657afbc" containerID="6e86456cba347d96aff3dc42fca741a1c2aa3e619b8b3445401c56e9ccbfd30a" exitCode=0
Oct 31 04:10:30.041193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.041171    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" event={"ID":"152455fe-94b2-43c8-b7dc-2c196657afbc","Type":"ContainerDied","Data":"6e86456cba347d96aff3dc42fca741a1c2aa3e619b8b3445401c56e9ccbfd30a"}
Oct 31 04:10:30.074277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.074252    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:10:30.205274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.203001    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\") pod \"152455fe-94b2-43c8-b7dc-2c196657afbc\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") "
Oct 31 04:10:30.205274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.203056    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\") pod \"152455fe-94b2-43c8-b7dc-2c196657afbc\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") "
Oct 31 04:10:30.205274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.203101    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\" (UniqueName: \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\") pod \"152455fe-94b2-43c8-b7dc-2c196657afbc\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") "
Oct 31 04:10:30.205274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.203317    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") pod \"152455fe-94b2-43c8-b7dc-2c196657afbc\" (UID: \"152455fe-94b2-43c8-b7dc-2c196657afbc\") "
Oct 31 04:10:30.221254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.213088    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca" pod "152455fe-94b2-43c8-b7dc-2c196657afbc" (UID: "152455fe-94b2-43c8-b7dc-2c196657afbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:30.221254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.216033    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v" pod "152455fe-94b2-43c8-b7dc-2c196657afbc" (UID: "152455fe-94b2-43c8-b7dc-2c196657afbc"). InnerVolumeSpecName "kube-api-access-cth9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:30.233444 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.230689    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs" pod "152455fe-94b2-43c8-b7dc-2c196657afbc" (UID: "152455fe-94b2-43c8-b7dc-2c196657afbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:30.233444 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.233332    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls" pod "152455fe-94b2-43c8-b7dc-2c196657afbc" (UID: "152455fe-94b2-43c8-b7dc-2c196657afbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:30.328608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.323554    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cth9v\" (UniqueName: \"kubernetes.io/projected/152455fe-94b2-43c8-b7dc-2c196657afbc-kube-api-access-cth9v\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:30.328608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.323596    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:30.328608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.323609    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152455fe-94b2-43c8-b7dc-2c196657afbc-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:30.328608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:30.323623    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152455fe-94b2-43c8-b7dc-2c196657afbc-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:31.051344 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:31.051316    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"
Oct 31 04:10:31.051833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:31.051354    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp" event={"ID":"152455fe-94b2-43c8-b7dc-2c196657afbc","Type":"ContainerDied","Data":"19705ca900a2f3056472db1f0b6c1379d38faa5323445141ff9d62deea63cd17"}
Oct 31 04:10:31.051833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:31.051428    5831 scope.go:122] "RemoveContainer" containerID="6e86456cba347d96aff3dc42fca741a1c2aa3e619b8b3445401c56e9ccbfd30a"
Oct 31 04:10:31.204061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:31.204016    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:10:31.249867 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:31.249832    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f1314-6c9d68c959-5rzxp"]
Oct 31 04:10:32.136545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:32.136500    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152455fe-94b2-43c8-b7dc-2c196657afbc" path="/var/lib/kubelet/pods/152455fe-94b2-43c8-b7dc-2c196657afbc/volumes"
Oct 31 04:10:33.196063 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:33.196019    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:10:33.196548 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:33.196298    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy" containerID="cri-o://73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813" gracePeriod=30
Oct 31 04:10:34.813907 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.813867    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:10:34.815209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815124    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8abfb9e6-a260-41df-aad2-05c6cb42299a" containerName="payload-processing"
Oct 31 04:10:34.815209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815156    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abfb9e6-a260-41df-aad2-05c6cb42299a" containerName="payload-processing"
Oct 31 04:10:34.815209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815174    5831 cpu_manager.go:414] "containerMap: removing container" podUID="152455fe-94b2-43c8-b7dc-2c196657afbc" containerName="maas-api"
Oct 31 04:10:34.815209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815184    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152455fe-94b2-43c8-b7dc-2c196657afbc" containerName="maas-api"
Oct 31 04:10:34.815487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815325    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abfb9e6-a260-41df-aad2-05c6cb42299a" containerName="payload-processing"
Oct 31 04:10:34.815487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.815343    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="152455fe-94b2-43c8-b7dc-2c196657afbc" containerName="maas-api"
Oct 31 04:10:34.821437 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.821414    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911461    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911523    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911617    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911657    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911685    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6cq\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911711    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.911830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911801    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.912114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911921    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.912114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.911968    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:34.972347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:34.956832    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e53098-openshift-default-dockercfg-c5hd9\""
Oct 31 04:10:35.066108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.016286    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.070848 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.045920    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.071057 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.070955    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.071057 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.071025    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.071764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.071709    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.071897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.071829    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.071897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.071863    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6cq\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.072026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.071917    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.072114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.072093    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.073311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.072329    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.073311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.072507    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.073311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.072816    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.073311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.072934    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.073311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.073052    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.078357 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.077958    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:10:35.098265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.097195    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.098265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.097754    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.135408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.135301    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6cq\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.185510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.167068    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\") pod \"e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") " pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.453314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.453267    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:35.911808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.911743    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:10:35.920519 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:10:35.919256    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c34cb3_a037_49e5_90e1_b02eb0b11bc7.slice/crio-9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e WatchSource:0}: Error finding container 9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e: Status 404 returned error can't find the container with id 9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e
Oct 31 04:10:35.924334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.924295    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:35.924455 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.924387    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:35.924455 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:35.924436    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:36.155584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:36.155229    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerStarted","Data":"bd6a7446a13645ded420789fbe9b9577b891a34bae0c07c20c8897cf40d4ce06"}
Oct 31 04:10:36.155584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:36.155304    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerStarted","Data":"9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e"}
Oct 31 04:10:36.241619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:36.240321    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podStartSLOduration=2.240302352 podStartE2EDuration="2.240302352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:36.236797225 +0000 UTC m=+3786.251358950" watchObservedRunningTime="2026-10-31 04:10:36.240302352 +0000 UTC m=+3786.254864061"
Oct 31 04:10:36.453648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:36.453604    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:37.460016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:37.459985    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:38.177957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.177922    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:38.185511 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.185481    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:10:38.528303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.528276    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:10:38.567016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.566980    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567040    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567094    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567126    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567159    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567207    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567287    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567318    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567334    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:38.567409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567371    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\") pod \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\" (UID: \"22dfd5a1-079f-40c6-ac0c-e9425090bce5\") "
Oct 31 04:10:38.567788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567622    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:38.567788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567636    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:38.567788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567647    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:38.567788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567652    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:38.568050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567960    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.568050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567984    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.568050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.567998    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.568050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.568013    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.568050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.568027    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.569814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.569791    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:38.569938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.569836    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:10:38.570330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.570311    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:38.570450 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.570333    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn" pod "22dfd5a1-079f-40c6-ac0c-e9425090bce5" (UID: "22dfd5a1-079f-40c6-ac0c-e9425090bce5"). InnerVolumeSpecName "kube-api-access-2z6kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:38.669279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.669241    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.669279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.669273    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.669279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.669282    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dfd5a1-079f-40c6-ac0c-e9425090bce5-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:38.669672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:38.669290    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6kn\" (UniqueName: \"kubernetes.io/projected/22dfd5a1-079f-40c6-ac0c-e9425090bce5-kube-api-access-2z6kn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:39.188575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.188498    5831 generic.go:356] "Generic (PLEG): container finished" podID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerID="73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813" exitCode=0
Oct 31 04:10:39.188768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.188701    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerDied","Data":"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"}
Oct 31 04:10:39.188768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.188757    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"
Oct 31 04:10:39.188930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.188773    5831 scope.go:122] "RemoveContainer" containerID="73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"
Oct 31 04:10:39.190676 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.188760    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2" event={"ID":"22dfd5a1-079f-40c6-ac0c-e9425090bce5","Type":"ContainerDied","Data":"5e4f1eed584e0543c46aa80fd8898b7cabf7807c5d094b96c25284a0bcb8dd15"}
Oct 31 04:10:39.204213 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.204189    5831 scope.go:122] "RemoveContainer" containerID="48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"
Oct 31 04:10:39.215478 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.215450    5831 scope.go:122] "RemoveContainer" containerID="73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"
Oct 31 04:10:39.215856 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:10:39.215832    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813\": container with ID starting with 73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813 not found: ID does not exist" containerID="73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"
Oct 31 04:10:39.215935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.215867    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813"} err="failed to get container status \"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813\": rpc error: code = NotFound desc = could not find container \"73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813\": container with ID starting with 73fded8cf17b78843f441b8042face3d145a20963aeef1de102a97e6006cf813 not found: ID does not exist"
Oct 31 04:10:39.215935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.215886    5831 scope.go:122] "RemoveContainer" containerID="48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"
Oct 31 04:10:39.216188 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:10:39.216160    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff\": container with ID starting with 48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff not found: ID does not exist" containerID="48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"
Oct 31 04:10:39.216261 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.216198    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff"} err="failed to get container status \"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff\": rpc error: code = NotFound desc = could not find container \"48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff\": container with ID starting with 48ac3257d0bae8b7c59ccb8a050ca579c9f3060f932cd74d7f64d0492e5a2cff not found: ID does not exist"
Oct 31 04:10:39.346430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.346392    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:10:39.395810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:39.395770    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1f1314-openshift-default-6b76475b9-spcj2"]
Oct 31 04:10:40.136391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:40.136360    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" path="/var/lib/kubelet/pods/22dfd5a1-079f-40c6-ac0c-e9425090bce5/volumes"
Oct 31 04:10:42.082806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:42.082773    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:10:42.083657 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:42.083623    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy" containerID="cri-o://e59b7b32036bd95d8e046a69f9d7355ba924919f4bab166f90d16a8600c0d1bd" gracePeriod=30
Oct 31 04:10:42.979508 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:10:42.979461    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:10:43.668809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.668767    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:10:43.669695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669655    5831 cpu_manager.go:414] "containerMap: removing container" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.669695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669691    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.669895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669719    5831 cpu_manager.go:414] "containerMap: removing container" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.669895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669724    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.669895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669824    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.669895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.669840    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dfd5a1-079f-40c6-ac0c-e9425090bce5" containerName="istio-proxy"
Oct 31 04:10:43.675327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.675308    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738055    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738331 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738162    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grwj\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738331 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738284    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738413    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738463    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738613    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738709    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738747    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.738878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.738854    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840641    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840698    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grwj\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840756    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840846    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840893    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.840997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.840966    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841024    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841048    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841092    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841283    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841479    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.841667 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.841611    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.842067 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.842013    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.842298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.842275    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.843666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.843642    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.843748 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.843642    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.863107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.863072    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:10:43.897163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.897091    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.897356 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.897232    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grwj\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\") pod \"maas-default-gateway-openshift-default-6886b96f9-cb4gc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:43.993269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:43.993220    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:44.354608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.337606    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:10:44.401105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.397602    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:10:44.401105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.397758    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.448863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.448824    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e53098\""
Oct 31 04:10:44.449061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.448899    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e53098-dockercfg-jkdj2\""
Oct 31 04:10:44.454913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.454882    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66wz\" (UniqueName: \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.455123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.454986    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.548456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.548409    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:10:44.551157 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:10:44.551129    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e9fbeb_0d6d_4a8b_9a52_7a822613e8dc.slice/crio-498f721701756b5ed559e0eef9b037bfb2630702fb97ee1aa6d05cd73fab5f5a WatchSource:0}: Error finding container 498f721701756b5ed559e0eef9b037bfb2630702fb97ee1aa6d05cd73fab5f5a: Status 404 returned error can't find the container with id 498f721701756b5ed559e0eef9b037bfb2630702fb97ee1aa6d05cd73fab5f5a
Oct 31 04:10:44.553903 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.553864    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:44.553994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.553948    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:44.553994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.553981    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:10:44.556500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.556469    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66wz\" (UniqueName: \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.556666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.556594    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.557320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.557298    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.609888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.609818    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66wz\" (UniqueName: \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\") pod \"payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:44.756004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:44.755975    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:45.146421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.146396    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:10:45.239720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.239680    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" event={"ID":"12d1c8c9-f286-4cd5-8863-fa926efb7d65","Type":"ContainerStarted","Data":"13f06aa32ae5fc915b239d9b15384176548bf006a5ed56b0665a455131976ef9"}
Oct 31 04:10:45.241476 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.241443    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" event={"ID":"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc","Type":"ContainerStarted","Data":"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"}
Oct 31 04:10:45.241643 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.241498    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" event={"ID":"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc","Type":"ContainerStarted","Data":"498f721701756b5ed559e0eef9b037bfb2630702fb97ee1aa6d05cd73fab5f5a"}
Oct 31 04:10:45.361084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.361021    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podStartSLOduration=2.361005065 podStartE2EDuration="2.361005065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:45.358523184 +0000 UTC m=+3795.373084896" watchObservedRunningTime="2026-10-31 04:10:45.361005065 +0000 UTC m=+3795.375566773"
Oct 31 04:10:45.994241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.994206    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:45.997728 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.997697    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:45.997996 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:45.997954    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:46.250503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:46.250404    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" event={"ID":"12d1c8c9-f286-4cd5-8863-fa926efb7d65","Type":"ContainerStarted","Data":"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"}
Oct 31 04:10:46.250729 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:46.250708    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:46.364204 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:46.364141    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" podStartSLOduration=2.364119208 podStartE2EDuration="2.364119208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:46.360014595 +0000 UTC m=+3796.374576303" watchObservedRunningTime="2026-10-31 04:10:46.364119208 +0000 UTC m=+3796.378680921"
Oct 31 04:10:47.005556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.003628    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:47.005556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.003693    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:47.261294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.260404    5831 generic.go:356] "Generic (PLEG): container finished" podID="098782ce-3828-401e-890e-a33f83cfca62" containerID="e59b7b32036bd95d8e046a69f9d7355ba924919f4bab166f90d16a8600c0d1bd" exitCode=0
Oct 31 04:10:47.261294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.260473    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerDied","Data":"e59b7b32036bd95d8e046a69f9d7355ba924919f4bab166f90d16a8600c0d1bd"}
Oct 31 04:10:47.261294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.260574    5831 scope.go:122] "RemoveContainer" containerID="5ad94f85d37db0df07b5ca5dff8ccd17eb62d194db5f2e9f7db905d893d2d617"
Oct 31 04:10:47.425625 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.425600    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:10:47.519683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519595    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.519683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.519906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519804    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.519906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519853    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.519906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519856    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:47.519906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519871    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.519906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519890    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.520216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519911    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.520216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519932    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.520216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.519984    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\") pod \"098782ce-3828-401e-890e-a33f83cfca62\" (UID: \"098782ce-3828-401e-890e-a33f83cfca62\") "
Oct 31 04:10:47.520216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520026    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:47.520458 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520408    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:47.520600 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520430    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:47.520682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520428    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.520682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520674    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.520791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.520726    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:47.522306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.522272    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:47.522578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.522516    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "kube-api-access-nfd4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:47.522841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.522812    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:10:47.523019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.522814    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token" pod "098782ce-3828-401e-890e-a33f83cfca62" (UID: "098782ce-3828-401e-890e-a33f83cfca62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:47.622277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622228    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622273    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfd4v\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-kube-api-access-nfd4v\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622290    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622306    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098782ce-3828-401e-890e-a33f83cfca62-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622320    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098782ce-3828-401e-890e-a33f83cfca62-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622335    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098782ce-3828-401e-890e-a33f83cfca62-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.622588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.622349    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098782ce-3828-401e-890e-a33f83cfca62-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:47.996979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.996942    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:47.997282 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:47.997252    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:48.270750 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:48.270660    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"
Oct 31 04:10:48.270750 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:48.270693    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd" event={"ID":"098782ce-3828-401e-890e-a33f83cfca62","Type":"ContainerDied","Data":"c9e0794d6d76b97a000abc716785e4f872a75a169c09938b0d546955c470699c"}
Oct 31 04:10:48.271302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:48.270752    5831 scope.go:122] "RemoveContainer" containerID="e59b7b32036bd95d8e046a69f9d7355ba924919f4bab166f90d16a8600c0d1bd"
Oct 31 04:10:48.424066 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:48.424020    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:10:48.427258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:48.427234    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5901fe-openshift-default-74f55d57c9-9q7hd"]
Oct 31 04:10:49.009557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:49.007933    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:49.009557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:49.008015    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:49.994556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:49.993956    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:49.994556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:49.994019    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:50.136482 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:50.136447    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098782ce-3828-401e-890e-a33f83cfca62" path="/var/lib/kubelet/pods/098782ce-3828-401e-890e-a33f83cfca62/volumes"
Oct 31 04:10:50.994670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:50.994628    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:50.995091 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:50.994697    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:51.993647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:51.993602    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cb4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 31 04:10:51.993871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:51.993673    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 31 04:10:53.091773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:53.091728    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:53.093347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:53.093325    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:53.094117 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:53.094095    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:10:57.264216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:57.264186    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:10:59.872115 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872082    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:10:59.872767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872752    5831 cpu_manager.go:414] "containerMap: removing container" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.872819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872768    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.872819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872784    5831 cpu_manager.go:414] "containerMap: removing container" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.872819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872790    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.872925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872884    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.872925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.872894    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="098782ce-3828-401e-890e-a33f83cfca62" containerName="istio-proxy"
Oct 31 04:10:59.876618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.876593    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:10:59.929044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.929008    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d247af-openshift-default-dockercfg-m8jkm\""
Oct 31 04:10:59.981314 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:10:59.981278    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:11:00.000734 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.000694    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.000935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.000789    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.000935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.000879    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zpp\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.000961    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.001094    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.001142    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.001173    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.001222    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.001391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.001289    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103011    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103080    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103112    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103167    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103229    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103278    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103342    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103369    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zpp\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103405    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103427    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.103817 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.103605    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.104247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.104223    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.104448 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.104334    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.104586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.104399    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.107571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.107552    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.108002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.107980    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.159605 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.158744    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.179564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.176134    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zpp\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\") pod \"e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") " pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.199780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.199736    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:00.684006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.683968    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:11:00.684800 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:11:00.684774    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663a1ddc_8f87_4209_a0e9_a6cd3bb72234.slice/crio-13d5d8fddba8546a1ae264bd1a24069e1e7cbfd51ae17cee1fc140f35da11120 WatchSource:0}: Error finding container 13d5d8fddba8546a1ae264bd1a24069e1e7cbfd51ae17cee1fc140f35da11120: Status 404 returned error can't find the container with id 13d5d8fddba8546a1ae264bd1a24069e1e7cbfd51ae17cee1fc140f35da11120
Oct 31 04:11:00.688185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.688145    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:00.688305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.688236    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:00.688305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:00.688277    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:01.399172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:01.399128    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerStarted","Data":"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"}
Oct 31 04:11:01.399724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:01.399196    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerStarted","Data":"13d5d8fddba8546a1ae264bd1a24069e1e7cbfd51ae17cee1fc140f35da11120"}
Oct 31 04:11:01.555104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:01.555040    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podStartSLOduration=2.5550214589999998 podStartE2EDuration="2.555021459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:01.553840841 +0000 UTC m=+3811.568402548" watchObservedRunningTime="2026-10-31 04:11:01.555021459 +0000 UTC m=+3811.569583168"
Oct 31 04:11:02.201157 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:02.201117    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:02.210104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:02.210070    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:02.410157 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:02.410119    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:02.411444 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:02.411416    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:11:03.906112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:03.906078    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:11:11.869500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:11.869457    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:11:11.870001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:11.869759    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" podUID="49e77eea-bf94-4685-aa6c-355f4910356b" containerName="maas-api" containerID="cri-o://bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259" gracePeriod=30
Oct 31 04:11:12.222688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.222657    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:11:12.333309 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.333273    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") pod \"49e77eea-bf94-4685-aa6c-355f4910356b\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") "
Oct 31 04:11:12.334204 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.334177    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\") pod \"49e77eea-bf94-4685-aa6c-355f4910356b\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") "
Oct 31 04:11:12.334461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.334438    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\" (UniqueName: \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\") pod \"49e77eea-bf94-4685-aa6c-355f4910356b\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") "
Oct 31 04:11:12.334850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.334822    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\") pod \"49e77eea-bf94-4685-aa6c-355f4910356b\" (UID: \"49e77eea-bf94-4685-aa6c-355f4910356b\") "
Oct 31 04:11:12.340225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.339919    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca" pod "49e77eea-bf94-4685-aa6c-355f4910356b" (UID: "49e77eea-bf94-4685-aa6c-355f4910356b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:12.437044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.436977    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e77eea-bf94-4685-aa6c-355f4910356b-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:12.437722 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.437640    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls" pod "49e77eea-bf94-4685-aa6c-355f4910356b" (UID: "49e77eea-bf94-4685-aa6c-355f4910356b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:12.445656 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.445614    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs" pod "49e77eea-bf94-4685-aa6c-355f4910356b" (UID: "49e77eea-bf94-4685-aa6c-355f4910356b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:12.445869 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.445845    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4" pod "49e77eea-bf94-4685-aa6c-355f4910356b" (UID: "49e77eea-bf94-4685-aa6c-355f4910356b"). InnerVolumeSpecName "kube-api-access-84tp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:12.500964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.500905    5831 generic.go:356] "Generic (PLEG): container finished" podID="49e77eea-bf94-4685-aa6c-355f4910356b" containerID="bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259" exitCode=0
Oct 31 04:11:12.501693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.501647    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" event={"ID":"49e77eea-bf94-4685-aa6c-355f4910356b","Type":"ContainerDied","Data":"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"}
Oct 31 04:11:12.521878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.501888    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt" event={"ID":"49e77eea-bf94-4685-aa6c-355f4910356b","Type":"ContainerDied","Data":"f9d2262e800263a6032dfc59663988156093e60e3c2943e3c14af9b7d51ab950"}
Oct 31 04:11:12.522172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.511667    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"
Oct 31 04:11:12.522580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.512900    5831 scope.go:122] "RemoveContainer" containerID="bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"
Oct 31 04:11:12.539193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.539019    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:12.539193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.539051    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tp4\" (UniqueName: \"kubernetes.io/projected/49e77eea-bf94-4685-aa6c-355f4910356b-kube-api-access-84tp4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:12.539193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.539063    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e77eea-bf94-4685-aa6c-355f4910356b-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:12.553395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.553286    5831 scope.go:122] "RemoveContainer" containerID="bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"
Oct 31 04:11:12.553796 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:12.553699    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259\": container with ID starting with bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259 not found: ID does not exist" containerID="bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"
Oct 31 04:11:12.553796 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.553741    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259"} err="failed to get container status \"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259\": rpc error: code = NotFound desc = could not find container \"bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259\": container with ID starting with bf430ea274ae3e4dec3aada4c22c3adcbcd1940cff5d93092328cad28853e259 not found: ID does not exist"
Oct 31 04:11:12.676480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.676424    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:11:12.680640 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:12.680362    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d253a5-64df697dd6-5wkdt"]
Oct 31 04:11:13.123571 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:13.123308    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:11:14.136566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:14.136497    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e77eea-bf94-4685-aa6c-355f4910356b" path="/var/lib/kubelet/pods/49e77eea-bf94-4685-aa6c-355f4910356b/volumes"
Oct 31 04:11:21.658730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:21.658694    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:11:21.659391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:21.658960    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy" containerID="cri-o://530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7" gracePeriod=30
Oct 31 04:11:25.009962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.009921    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:11:25.010654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.010636    5831 cpu_manager.go:414] "containerMap: removing container" podUID="49e77eea-bf94-4685-aa6c-355f4910356b" containerName="maas-api"
Oct 31 04:11:25.010714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.010655    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e77eea-bf94-4685-aa6c-355f4910356b" containerName="maas-api"
Oct 31 04:11:25.010791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.010782    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e77eea-bf94-4685-aa6c-355f4910356b" containerName="maas-api"
Oct 31 04:11:25.014600 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.014578    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.063042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.063008    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e2a54d-openshift-default-dockercfg-s4hdf\""
Oct 31 04:11:25.122017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.121969    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:11:25.229575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.209144    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.229919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.229894    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.230123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.230105    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.230346 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.230312    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.230481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.230468    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.230807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.230785    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.230981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.230958    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwgq\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.231079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.231012    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.231079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.231046    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.332891 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.332802    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.332891 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.332856    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwgq\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.332983    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333036    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333109    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333157    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333195    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333205    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333296    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333326    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333642 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.333642 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.333381    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.338659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.334163    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.338659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.334311    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.338659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.336204    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.365059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.365023    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.389872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.389837    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwgq\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.422875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.422843    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\") pod \"e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") " pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:25.630030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:25.629941    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:26.075814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.075780    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:11:26.077308 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:11:26.077273    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784e2fcf_7f87_4d52_9569_92918e2caf8d.slice/crio-395f2cfcdf2a49bf561f2f6a4b11978bf55f34380e49bac2589a5d42144ebd0d WatchSource:0}: Error finding container 395f2cfcdf2a49bf561f2f6a4b11978bf55f34380e49bac2589a5d42144ebd0d: Status 404 returned error can't find the container with id 395f2cfcdf2a49bf561f2f6a4b11978bf55f34380e49bac2589a5d42144ebd0d
Oct 31 04:11:26.107812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.106742    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:26.107812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.106824    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:26.107812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.106872    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:26.724586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.724507    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerStarted","Data":"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"}
Oct 31 04:11:26.724810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.724595    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerStarted","Data":"395f2cfcdf2a49bf561f2f6a4b11978bf55f34380e49bac2589a5d42144ebd0d"}
Oct 31 04:11:26.835618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:26.835554    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podStartSLOduration=2.835519926 podStartE2EDuration="2.835519926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:26.832936858 +0000 UTC m=+3836.847498560" watchObservedRunningTime="2026-10-31 04:11:26.835519926 +0000 UTC m=+3836.850081633"
Oct 31 04:11:27.099116 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.098759    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:11:27.151480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151447    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151502    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151585    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151680    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151717    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151792    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:27.151990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151814    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151853    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151897    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\") pod \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\" (UID: \"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd\") "
Oct 31 04:11:27.151990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151978    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:27.152251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.151993    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:27.152251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152179    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:27.152369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152277    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:27.152610 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152578    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.152735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152613    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.152735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152627    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.152735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152640    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.152735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.152653    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.154369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.154340    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:27.154726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.154703    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "kube-api-access-2599l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:27.154878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.154841    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:27.155170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.155145    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo" pod "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" (UID: "eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:11:27.254166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.254124    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-kube-api-access-2599l\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.254166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.254160    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.254166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.254169    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.254422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.254178    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:27.630658 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.630559    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:27.637135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.637103    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:27.736022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.735974    5831 generic.go:356] "Generic (PLEG): container finished" podID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerID="530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7" exitCode=0
Oct 31 04:11:27.736233 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.736043    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"
Oct 31 04:11:27.736233 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.736185    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerDied","Data":"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"}
Oct 31 04:11:27.736382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.736244    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw" event={"ID":"eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd","Type":"ContainerDied","Data":"6c60c59e85713639c0e14aa6f5317a1e9dc02aa2907814182fbcf0b157c70ad4"}
Oct 31 04:11:27.736382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.736276    5831 scope.go:122] "RemoveContainer" containerID="530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"
Oct 31 04:11:27.736656 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.736637    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:27.742795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.742769    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:11:27.758618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.758592    5831 scope.go:122] "RemoveContainer" containerID="5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"
Oct 31 04:11:27.771127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.771101    5831 scope.go:122] "RemoveContainer" containerID="530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"
Oct 31 04:11:27.771495 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:27.771472    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7\": container with ID starting with 530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7 not found: ID does not exist" containerID="530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"
Oct 31 04:11:27.771602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.771505    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7"} err="failed to get container status \"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7\": rpc error: code = NotFound desc = could not find container \"530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7\": container with ID starting with 530d26093395394edab489d705cdb8a933b4f35da4841a75e62047cbdc4c09a7 not found: ID does not exist"
Oct 31 04:11:27.771602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.771539    5831 scope.go:122] "RemoveContainer" containerID="5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"
Oct 31 04:11:27.771869 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:27.771829    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f\": container with ID starting with 5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f not found: ID does not exist" containerID="5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"
Oct 31 04:11:27.771869 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:27.771859    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f"} err="failed to get container status \"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f\": rpc error: code = NotFound desc = could not find container \"5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f\": container with ID starting with 5a318126d5e2323d67d5a0ccf90f965b8a5efb70ffe073d3e96ce9ead3bcee0f not found: ID does not exist"
Oct 31 04:11:28.017170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:28.017125    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:11:28.021545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:28.021496    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d253a5-openshift-default-875544d77-76bvw"]
Oct 31 04:11:28.136751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:28.136720    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" path="/var/lib/kubelet/pods/eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd/volumes"
Oct 31 04:11:33.611716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.611671    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:11:33.612389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.612354    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:11:33.682137 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.682096    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:11:33.683362 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.683329    5831 cpu_manager.go:414] "containerMap: removing container" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.683575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.683557    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.683811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.683795    5831 cpu_manager.go:414] "containerMap: removing container" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.683927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.683915    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.684311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.684292    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.684430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.684419    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeaa8ae1-2b7a-4f47-91e1-e3535ede5bdd" containerName="istio-proxy"
Oct 31 04:11:33.731773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.731734    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:11:33.738556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.732144    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.792841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.792790    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e2a54d\""
Oct 31 04:11:33.793192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.793148    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e2a54d-dockercfg-bdccn\""
Oct 31 04:11:33.811814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.811775    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.811994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.811867    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlsn\" (UniqueName: \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.913144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.913048    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlsn\" (UniqueName: \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.913659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.913341    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.914332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.914310    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:33.965790 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:33.965752    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlsn\" (UniqueName: \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\") pod \"payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:34.085579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.082963    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:34.486338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.486309    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:11:34.488907 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:11:34.488866    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78fa607_b3f6_44b3_8bfe_4dc3880948cb.slice/crio-a1a0b3301fed977a5beee12f647294f1dae53cf88022de0e5f4d0a11c430bcc1 WatchSource:0}: Error finding container a1a0b3301fed977a5beee12f647294f1dae53cf88022de0e5f4d0a11c430bcc1: Status 404 returned error can't find the container with id a1a0b3301fed977a5beee12f647294f1dae53cf88022de0e5f4d0a11c430bcc1
Oct 31 04:11:34.805919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.805817    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" event={"ID":"c78fa607-b3f6-44b3-8bfe-4dc3880948cb","Type":"ContainerStarted","Data":"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"}
Oct 31 04:11:34.805919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.805886    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" event={"ID":"c78fa607-b3f6-44b3-8bfe-4dc3880948cb","Type":"ContainerStarted","Data":"a1a0b3301fed977a5beee12f647294f1dae53cf88022de0e5f4d0a11c430bcc1"}
Oct 31 04:11:34.806458 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.805930    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:34.913086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:34.913026    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" podStartSLOduration=1.913010087 podStartE2EDuration="1.913010087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:34.910225728 +0000 UTC m=+3844.924787440" watchObservedRunningTime="2026-10-31 04:11:34.913010087 +0000 UTC m=+3844.927571796"
Oct 31 04:11:35.814856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.814818    5831 generic.go:356] "Generic (PLEG): container finished" podID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerID="e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe" exitCode=0
Oct 31 04:11:35.815393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.814891    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerDied","Data":"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"}
Oct 31 04:11:35.815393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.815330    5831 scope.go:122] "RemoveContainer" containerID="e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"
Oct 31 04:11:35.816930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.816895    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:35.817062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.816966    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:35.817062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:35.816998    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:11:36.824029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:36.823993    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerStarted","Data":"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"}
Oct 31 04:11:37.485554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:37.482926    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:11:37.485554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:37.483048    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 31 04:11:37.485554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:37.483090    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 31 04:11:38.483975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:38.483936    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 31 04:11:38.484617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:38.484584    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 31 04:11:39.484019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:39.483974    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 31 04:11:39.484689 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:39.484656    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 31 04:11:40.007369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.007328    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:11:40.013558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.013516    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.053304 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.053273    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:11:40.105159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.105121    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-83f052-serving-cert\""
Oct 31 04:11:40.121653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.121615    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x555\" (UniqueName: \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.121854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.121748    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.121854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.121830    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.121854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.121849    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.223357 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.223320    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x555\" (UniqueName: \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.223690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.223458    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.223690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.223592    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.223690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.223623    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.223976 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:40.223754    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-serving-cert: secret "maas-api-e2e-inf-a-83f052-serving-cert" not found
Oct 31 04:11:40.223976 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:40.223817    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls podName:b2b081d0-dfb9-4739-9366-fea4bc734067 nodeName:}" failed. No retries permitted until 2026-10-31 04:11:40.723797125 +0000 UTC m=+3850.738358829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls") pod "maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" (UID: "b2b081d0-dfb9-4739-9366-fea4bc734067") : secret "maas-api-e2e-inf-a-83f052-serving-cert" not found
Oct 31 04:11:40.224612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.224521    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.226733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.226709    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.278004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.277932    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x555\" (UniqueName: \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.485555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.481919    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 31 04:11:40.485555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.481981    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 31 04:11:40.751513 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.751473    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.754150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.754128    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") pod \"maas-api-e2e-inf-a-83f052-59c596497d-2pbdg\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:40.928373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:40.928331    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:41.336435 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:41.336347    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:11:41.338516 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:11:41.338488    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b081d0_dfb9_4739_9366_fea4bc734067.slice/crio-132bf7a9e6a5130834b65297f61553a59d7506bad5dc9e3f62b649692f542b2b WatchSource:0}: Error finding container 132bf7a9e6a5130834b65297f61553a59d7506bad5dc9e3f62b649692f542b2b: Status 404 returned error can't find the container with id 132bf7a9e6a5130834b65297f61553a59d7506bad5dc9e3f62b649692f542b2b
Oct 31 04:11:41.340130 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:41.340112    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:11:41.482993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:41.482955    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 31 04:11:41.483295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:41.483260    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 31 04:11:41.864103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:41.864061    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" event={"ID":"b2b081d0-dfb9-4739-9366-fea4bc734067","Type":"ContainerStarted","Data":"132bf7a9e6a5130834b65297f61553a59d7506bad5dc9e3f62b649692f542b2b"}
Oct 31 04:11:42.485338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.485305    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:11:42.485860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.485840    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:11:42.486742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.486722    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:11:42.885520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.885417    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" event={"ID":"b2b081d0-dfb9-4739-9366-fea4bc734067","Type":"ContainerStarted","Data":"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"}
Oct 31 04:11:42.885520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.885551    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:42.983763 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:11:42.983725    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:11:42.991259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:42.991209    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" podStartSLOduration=3.164504456 podStartE2EDuration="3.991194278s" totalImagesPullingTime="826.689822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:42.990114305 +0000 UTC m=+3853.004676012" watchObservedRunningTime="2026-10-31 04:11:42.991194278 +0000 UTC m=+3853.005755985"
Oct 31 04:11:45.818834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:45.818805    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:11:48.900659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:48.900630    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:11:56.213217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:11:56.213173    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:12:12.980872 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:12:12.980767    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:12:25.291792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:25.291757    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:12:36.364930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.364897    5831 generic.go:356] "Generic (PLEG): container finished" podID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerID="bd6a7446a13645ded420789fbe9b9577b891a34bae0c07c20c8897cf40d4ce06" exitCode=0
Oct 31 04:12:36.365387 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.364977    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerDied","Data":"bd6a7446a13645ded420789fbe9b9577b891a34bae0c07c20c8897cf40d4ce06"}
Oct 31 04:12:36.365431 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.365390    5831 scope.go:122] "RemoveContainer" containerID="bd6a7446a13645ded420789fbe9b9577b891a34bae0c07c20c8897cf40d4ce06"
Oct 31 04:12:36.367043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.367018    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:36.367101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.367088    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:36.367138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:36.367122    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:37.374129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:37.374086    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerStarted","Data":"c3ed032fa5fe04d4eaa9aa6c89a5c43651fa42ba3d6f949b63f641af3cf1ceec"}
Oct 31 04:12:37.454391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:37.454356    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:12:37.455480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:37.455458    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:37.455559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:37.455515    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:38.178352 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.178321    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:12:38.454695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.454603    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:38.454695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.454659    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:38.548649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.548621    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:12:38.550567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.550543    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:12:38.559552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.559515    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:12:38.561277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.561256    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:12:38.563132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.563114    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:12:38.565811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:38.565775    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:12:39.454215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:39.454174    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:39.454413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:39.454245    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:40.010563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:39.990511    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:12:40.039966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.039936    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.219563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.174298    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-243a29-openshift-default-dockercfg-zm8fm\""
Oct 31 04:12:40.256656 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.221795    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.257227 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.257200    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.285780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.285681    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.286121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.286092    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.286298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.286281    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swj9\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.286420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.286406    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.292128 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.292102    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.298554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.292303    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.298925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.298896    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.398494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.398460    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:12:40.408402 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.408334    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.408717 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.408683    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.408957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.408938    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.409127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.409110    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swj9\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.409284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.409266    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.409439 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.409423    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.409663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.409645    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.409975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.409956    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.410252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.410216    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.411041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.411020    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.414623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.413273    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.414623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.413977    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.415570 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.415550    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.416027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.415995    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.443661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.443632    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.448900 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.448848    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.454770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.454709    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:40.455083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.455027    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:40.469464 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.469402    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.469678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.469659    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swj9\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\") pod \"e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") " pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:40.592731 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:40.592642    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:41.036910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.036878    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:12:41.037284 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:12:41.037155    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb0604b_7877_4362_b83d_a3eb3e62cb9a.slice/crio-125d9bb0e1704f23cdeaa34c2aab9ba94406782ce3ed8ba1bda461a1e7aa61e8 WatchSource:0}: Error finding container 125d9bb0e1704f23cdeaa34c2aab9ba94406782ce3ed8ba1bda461a1e7aa61e8: Status 404 returned error can't find the container with id 125d9bb0e1704f23cdeaa34c2aab9ba94406782ce3ed8ba1bda461a1e7aa61e8
Oct 31 04:12:41.041980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.039780    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:41.041980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.039845    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:41.041980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.039873    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:41.448071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.448036    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerStarted","Data":"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"}
Oct 31 04:12:41.448259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.448098    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerStarted","Data":"125d9bb0e1704f23cdeaa34c2aab9ba94406782ce3ed8ba1bda461a1e7aa61e8"}
Oct 31 04:12:41.453641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.453616    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:41.453724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.453669    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:41.558465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.558399    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podStartSLOduration=2.55838455 podStartE2EDuration="2.55838455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:41.556583587 +0000 UTC m=+3911.571145294" watchObservedRunningTime="2026-10-31 04:12:41.55838455 +0000 UTC m=+3911.572946258"
Oct 31 04:12:41.597555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:41.596634    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:42.455562 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:42.455328    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body=
Oct 31 04:12:42.455562 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:42.455410    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused"
Oct 31 04:12:42.628256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:42.627890    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:42.990878 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:12:42.990832    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:12:43.464857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:43.464827    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:12:43.466727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:43.466689    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:12:43.480162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:43.480125    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:43.480927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:43.480905    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:12:56.650634 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.650519    5831 generic.go:356] "Generic (PLEG): container finished" podID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerID="d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480" exitCode=0
Oct 31 04:12:56.650634 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.650592    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerDied","Data":"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"}
Oct 31 04:12:56.651196 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.651027    5831 scope.go:122] "RemoveContainer" containerID="d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"
Oct 31 04:12:56.652827 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.652799    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:56.652891 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.652871    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:56.652929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:56.652904    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:12:57.660675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:57.660627    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerStarted","Data":"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"}
Oct 31 04:12:58.209719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:58.209681    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:12:58.211393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:58.211354    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 31 04:12:58.211590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:58.211417    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 31 04:12:59.205022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:59.204972    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 31 04:12:59.205576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:12:59.205040    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 31 04:13:00.202555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:00.200890    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 31 04:13:00.202555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:00.200953    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 31 04:13:01.203364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:01.202915    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 31 04:13:01.203364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:01.202980    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 31 04:13:02.203138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:02.203102    5831 patch_prober.go:29] interesting pod/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body=
Oct 31 04:13:02.203546 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:02.203494    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused"
Oct 31 04:13:02.410028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:02.409992    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:13:03.206340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:03.206312    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:13:03.207662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:03.207635    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:13:12.982450 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:13:12.982406    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:13:16.891540 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:16.891501    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:13:26.847438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:26.847401    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:13:26.851306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:26.851282    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:26.904429 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:26.904369    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:13:26.972555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:26.948002    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f30f2-dockercfg-4ttqg\""
Oct 31 04:13:26.973074 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:26.973040    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f30f2\""
Oct 31 04:13:27.128611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.115582    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4tzx\" (UniqueName: \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.135486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.135452    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.151935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.151387    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:13:27.158627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.158598    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.162675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.162600    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:13:27.237612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.237576    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.237803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.237695    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flq6m\" (UniqueName: \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.237803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.237726    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.237803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.237779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4tzx\" (UniqueName: \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.238559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.238508    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.290858 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.290822    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4tzx\" (UniqueName: \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\") pod \"payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.339925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.339881    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flq6m\" (UniqueName: \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.340122 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.339959    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.340675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.340654    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.393149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.393071    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flq6m\" (UniqueName: \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\") pod \"payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.466854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.466812    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:27.498926 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.498862    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:27.860761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.857139    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:13:27.860761 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:13:27.858491    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b9f74f_fb2a_4c2d_a1a4_beac82ecc6b8.slice/crio-8c97095a956f427a3354b595d95bdd73492e157a0a8c44f4a8d6bc1e981908d0 WatchSource:0}: Error finding container 8c97095a956f427a3354b595d95bdd73492e157a0a8c44f4a8d6bc1e981908d0: Status 404 returned error can't find the container with id 8c97095a956f427a3354b595d95bdd73492e157a0a8c44f4a8d6bc1e981908d0
Oct 31 04:13:27.908592 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.908564    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:13:27.914073 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:13:27.914038    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ce1685_c126_4d48_b9a0_d42efec02ec8.slice/crio-3a62ddf93b467495af40710262732f3e5fe9e303510bd6f866ec8050fe2e9061 WatchSource:0}: Error finding container 3a62ddf93b467495af40710262732f3e5fe9e303510bd6f866ec8050fe2e9061: Status 404 returned error can't find the container with id 3a62ddf93b467495af40710262732f3e5fe9e303510bd6f866ec8050fe2e9061
Oct 31 04:13:27.944219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.944175    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": read tcp 10.132.0.2:39882->10.132.1.78:15021: read: connection reset by peer" start-of-body=
Oct 31 04:13:27.944343 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:27.944234    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": read tcp 10.132.0.2:39882->10.132.1.78:15021: read: connection reset by peer"
Oct 31 04:13:28.130516 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.130133    5831 generic.go:356] "Generic (PLEG): container finished" podID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerID="07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2" exitCode=0
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204487    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204573    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerDied","Data":"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"}
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204612    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" event={"ID":"24ce1685-c126-4d48-b9a0-d42efec02ec8","Type":"ContainerStarted","Data":"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"}
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204634    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" event={"ID":"24ce1685-c126-4d48-b9a0-d42efec02ec8","Type":"ContainerStarted","Data":"3a62ddf93b467495af40710262732f3e5fe9e303510bd6f866ec8050fe2e9061"}
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204656    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" event={"ID":"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8","Type":"ContainerStarted","Data":"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"}
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204686    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.204706    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" event={"ID":"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8","Type":"ContainerStarted","Data":"8c97095a956f427a3354b595d95bdd73492e157a0a8c44f4a8d6bc1e981908d0"}
Oct 31 04:13:28.206627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.205314    5831 scope.go:122] "RemoveContainer" containerID="07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"
Oct 31 04:13:28.216112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.207926    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:13:28.216112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.208008    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:13:28.216112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.208046    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:13:28.261870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.261823    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" podStartSLOduration=2.261808883 podStartE2EDuration="2.261808883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:28.258709206 +0000 UTC m=+3958.273270911" watchObservedRunningTime="2026-10-31 04:13:28.261808883 +0000 UTC m=+3958.276370590"
Oct 31 04:13:28.408555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:28.404748    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" podStartSLOduration=2.404715802 podStartE2EDuration="2.404715802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:28.372980357 +0000 UTC m=+3958.387542083" watchObservedRunningTime="2026-10-31 04:13:28.404715802 +0000 UTC m=+3958.419277511"
Oct 31 04:13:29.164318 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:29.164278    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerStarted","Data":"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"}
Oct 31 04:13:29.630693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:29.630656    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:13:29.630913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:29.630883    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:29.630971 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:29.630940    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:30.631216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:30.631164    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:30.631786 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:30.631264    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:31.631087 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:31.631042    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:31.631320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:31.631099    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:32.631111 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:32.631071    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:32.631280 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:32.631153    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:33.630788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:33.630739    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:33.631316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:33.630812    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:34.630593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:34.630553    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Oct 31 04:13:34.630813 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:34.630623    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Oct 31 04:13:35.635497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:35.635464    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:13:35.636009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:35.635991    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:13:35.636661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:35.636643    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:13:39.168235 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:39.168208    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:13:39.168671 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:39.168271    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:13:42.980388 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:13:42.980341    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:13:44.269556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.267105    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:13:44.285109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.282648    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.286105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.286064    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:13:44.367375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.367333    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.367602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.367460    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4zw\" (UniqueName: \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.376669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.376632    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e679fe\""
Oct 31 04:13:44.376669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.376646    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e679fe-dockercfg-49tzp\""
Oct 31 04:13:44.469161 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.469129    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.469364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.469218    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4zw\" (UniqueName: \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.470010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.469988    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.522644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.522566    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4zw\" (UniqueName: \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\") pod \"payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.601972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.601938    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:44.978798 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:13:44.978764    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b0775f_f340_4aae_b789_58c1ed170160.slice/crio-cef9814efdffb8447ad1ffcac1be1ef75a39a3f5a589698f641fa36efebc0e61 WatchSource:0}: Error finding container cef9814efdffb8447ad1ffcac1be1ef75a39a3f5a589698f641fa36efebc0e61: Status 404 returned error can't find the container with id cef9814efdffb8447ad1ffcac1be1ef75a39a3f5a589698f641fa36efebc0e61
Oct 31 04:13:44.979192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:44.979164    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:13:45.330072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:45.329968    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" event={"ID":"f9b0775f-f340-4aae-b789-58c1ed170160","Type":"ContainerStarted","Data":"9a41ed38fd4de855264a73b5d3e15425af09771d7f1c3cf6b7ba4278602dbd5c"}
Oct 31 04:13:45.330072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:45.330026    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" event={"ID":"f9b0775f-f340-4aae-b789-58c1ed170160","Type":"ContainerStarted","Data":"cef9814efdffb8447ad1ffcac1be1ef75a39a3f5a589698f641fa36efebc0e61"}
Oct 31 04:13:45.330072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:45.330062    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:13:45.435167 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:45.435116    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" podStartSLOduration=1.435092898 podStartE2EDuration="1.435092898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:45.434931099 +0000 UTC m=+3975.449492809" watchObservedRunningTime="2026-10-31 04:13:45.435092898 +0000 UTC m=+3975.449654607"
Oct 31 04:13:56.341335 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:13:56.341289    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:14:12.980016 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:14:12.979975    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:14:34.776725 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:34.776680    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:14:36.107128 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:36.107087    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:14:39.777422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:39.777358    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" podUID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" containerName="payload-processing" containerID="cri-o://19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760" gracePeriod=25
Oct 31 04:14:40.141086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.141062    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:14:40.231973 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.231119    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\") pod \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") "
Oct 31 04:14:40.231973 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.231323    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\" (UniqueName: \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\") pod \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\" (UID: \"c78fa607-b3f6-44b3-8bfe-4dc3880948cb\") "
Oct 31 04:14:40.235813 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.235772    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume" pod "c78fa607-b3f6-44b3-8bfe-4dc3880948cb" (UID: "c78fa607-b3f6-44b3-8bfe-4dc3880948cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:40.256812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.256740    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn" pod "c78fa607-b3f6-44b3-8bfe-4dc3880948cb" (UID: "c78fa607-b3f6-44b3-8bfe-4dc3880948cb"). InnerVolumeSpecName "kube-api-access-jxlsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:40.334346 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.334305    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:40.334346 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.334335    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlsn\" (UniqueName: \"kubernetes.io/projected/c78fa607-b3f6-44b3-8bfe-4dc3880948cb-kube-api-access-jxlsn\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:40.871826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.871789    5831 generic.go:356] "Generic (PLEG): container finished" podID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" containerID="19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760" exitCode=0
Oct 31 04:14:40.872317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.871838    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"
Oct 31 04:14:40.872317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.871858    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" event={"ID":"c78fa607-b3f6-44b3-8bfe-4dc3880948cb","Type":"ContainerDied","Data":"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"}
Oct 31 04:14:40.872317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.871900    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f" event={"ID":"c78fa607-b3f6-44b3-8bfe-4dc3880948cb","Type":"ContainerDied","Data":"a1a0b3301fed977a5beee12f647294f1dae53cf88022de0e5f4d0a11c430bcc1"}
Oct 31 04:14:40.872317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.871921    5831 scope.go:122] "RemoveContainer" containerID="19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"
Oct 31 04:14:40.916227 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.916199    5831 scope.go:122] "RemoveContainer" containerID="19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"
Oct 31 04:14:40.916682 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:14:40.916655    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760\": container with ID starting with 19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760 not found: ID does not exist" containerID="19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"
Oct 31 04:14:40.916777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:40.916690    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760"} err="failed to get container status \"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760\": rpc error: code = NotFound desc = could not find container \"19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760\": container with ID starting with 19574bfdddd96a29406201c3fd2cd988b9f96908d8bd7f89c5f6535398eec760 not found: ID does not exist"
Oct 31 04:14:41.026691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:41.026658    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:14:41.077393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:41.076435    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e2a54d-7b99f6fbbc-8sz4f"]
Oct 31 04:14:42.137811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:42.137778    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" path="/var/lib/kubelet/pods/c78fa607-b3f6-44b3-8bfe-4dc3880948cb/volumes"
Oct 31 04:14:42.988037 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:14:42.987997    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:14:44.521768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:44.521711    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:14:44.522305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:44.521819    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:14:46.957471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.957440    5831 generic.go:356] "Generic (PLEG): container finished" podID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerID="3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a" exitCode=0
Oct 31 04:14:46.957865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.957518    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerDied","Data":"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"}
Oct 31 04:14:46.958101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.958085    5831 scope.go:122] "RemoveContainer" containerID="3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"
Oct 31 04:14:46.959821 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.959796    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:14:46.959905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.959883    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:14:46.959946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:46.959923    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:14:47.967114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:47.967077    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerStarted","Data":"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"}
Oct 31 04:14:48.595024 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:48.594986    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:14:48.595438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:48.595341    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 31 04:14:48.595669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:48.595646    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 31 04:14:49.594001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:49.593960    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 31 04:14:49.594692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:49.594658    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 31 04:14:50.598555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:50.597939    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 31 04:14:50.598555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:50.598003    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 31 04:14:51.601555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:51.601019    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 31 04:14:51.601555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:51.601100    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 31 04:14:52.599962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:52.599922    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body=
Oct 31 04:14:52.600281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:52.600254    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused"
Oct 31 04:14:53.598215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:53.598178    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:14:53.598783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:53.598763    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:14:53.599568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:53.599512    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:14:54.557758 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.557711    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:14:54.558491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.558475    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" containerName="payload-processing"
Oct 31 04:14:54.558561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.558492    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" containerName="payload-processing"
Oct 31 04:14:54.558617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.558607    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78fa607-b3f6-44b3-8bfe-4dc3880948cb" containerName="payload-processing"
Oct 31 04:14:54.562435 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.562414    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.615090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.615058    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:14:54.650152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.650112    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.650336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.650161    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.650336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.650200    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.650446 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.650378    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkq2\" (UniqueName: \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.655711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.655680    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6f30f2-serving-cert\""
Oct 31 04:14:54.752091 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.752054    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.752091 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.752089    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.752326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.752112    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.752326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.752164    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkq2\" (UniqueName: \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.752326 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:14:54.752214    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-serving-cert: secret "maas-api-e2e-inf-b-6f30f2-serving-cert" not found
Oct 31 04:14:54.752326 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:14:54.752293    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls podName:8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb nodeName:}" failed. No retries permitted until 2026-10-31 04:14:55.252276354 +0000 UTC m=+4045.266838044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls") pod "maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" (UID: "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb") : secret "maas-api-e2e-inf-b-6f30f2-serving-cert" not found
Oct 31 04:14:54.752871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.752853    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.754713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.754691    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:54.843616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:54.820608    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkq2\" (UniqueName: \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:55.260648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:55.260605    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:55.263388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:55.263362    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:55.477273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:55.477238    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:55.851978 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:55.851949    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:14:55.853664 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:14:55.853636    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8babf2_b3b3_4018_a6bd_c2ef4fe222cb.slice/crio-18d0bb273c4e041ad875f6c42712a02d7917729fd1429aa4af200dafbc2bf0f8 WatchSource:0}: Error finding container 18d0bb273c4e041ad875f6c42712a02d7917729fd1429aa4af200dafbc2bf0f8: Status 404 returned error can't find the container with id 18d0bb273c4e041ad875f6c42712a02d7917729fd1429aa4af200dafbc2bf0f8
Oct 31 04:14:56.063191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:56.063150    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" event={"ID":"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb","Type":"ContainerStarted","Data":"18d0bb273c4e041ad875f6c42712a02d7917729fd1429aa4af200dafbc2bf0f8"}
Oct 31 04:14:57.073312 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:57.073220    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" event={"ID":"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb","Type":"ContainerStarted","Data":"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"}
Oct 31 04:14:57.073312 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:57.073291    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:14:57.184355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:14:57.184300    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" podStartSLOduration=2.227682082 podStartE2EDuration="3.184285292s" totalImagesPullingTime="956.60321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:57.180821913 +0000 UTC m=+4047.195383620" watchObservedRunningTime="2026-10-31 04:14:57.184285292 +0000 UTC m=+4047.198847000"
Oct 31 04:15:00.165324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.165236    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:00.182378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.182340    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:00.182594 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.182462    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.187619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.187592    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:00.187771 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.187684    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.192542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.192455    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.213327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.213300    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:00.219164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.219140    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:00.219299 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.219279    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.224577 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.224515    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:00.224719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.224637    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.229348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.229320    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.236122 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.236100    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:00.240442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.240423    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.258495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.258469    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:00.262286 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.262263    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:00.265497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.265474    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:00.308009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.307973    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:00.332742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.332710    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:15:00.338376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338341    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.338587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338437    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.338587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338502    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.338587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338569    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzdx\" (UniqueName: \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.338764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338611    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.338852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338827    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn62w\" (UniqueName: \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.338904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338872    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmn7\" (UniqueName: \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.338985 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.338950    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5sbp\" (UniqueName: \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.339031 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339007    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.339083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339055    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6l2\" (UniqueName: \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.339131 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339093    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvmb\" (UniqueName: \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.339184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339169    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.339238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339213    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2p2\" (UniqueName: \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.339368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.339350    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.350719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.350679    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:00.369731 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.369694    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:00.406679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.406644    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:00.440270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440226    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.440270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440273    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzdx\" (UniqueName: \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440301    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440424    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn62w\" (UniqueName: \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440451    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmn7\" (UniqueName: \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440482    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5sbp\" (UniqueName: \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440506    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.440582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440575    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6l2\" (UniqueName: \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440614    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvmb\" (UniqueName: \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440691    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440736    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2p2\" (UniqueName: \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440794    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440842    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.440922 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.440877    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.441257 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441120    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.441257 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441235    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.441710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441686    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.441923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441898    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.442003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441977    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.442071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.441996    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.442295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.442275    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.495369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.495338    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6l2\" (UniqueName: \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\") pod \"maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.495580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.495440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5sbp\" (UniqueName: \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.495580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.495493    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvmb\" (UniqueName: \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\") pod \"maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.496179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.496154    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn62w\" (UniqueName: \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.496256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.496231    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzdx\" (UniqueName: \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\") pod \"maas-api-key-cleanup-29890335-g9xtq\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.496256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.496229    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmn7\" (UniqueName: \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.501258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.501227    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2p2\" (UniqueName: \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\") pod \"maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.503970 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.503952    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:00.511583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.511558    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:00.518814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.518772    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:00.536582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.535516    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:00.544445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.544401    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:00.554276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.554249    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:00.560923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.560757    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:00.920253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:00.920213    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:00.926022 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:00.925991    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9a34cf_a392_476d_97f4_db31bec14cb6.slice/crio-bcd719e68dcd6ac6d0ab879c00f00833ddad24a4ca2183eda4d0c1a6a78a00d8 WatchSource:0}: Error finding container bcd719e68dcd6ac6d0ab879c00f00833ddad24a4ca2183eda4d0c1a6a78a00d8: Status 404 returned error can't find the container with id bcd719e68dcd6ac6d0ab879c00f00833ddad24a4ca2183eda4d0c1a6a78a00d8
Oct 31 04:15:01.004838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.004803    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:01.006857 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.006829    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9f4d36_2477_4f1b_b40f_8e46fe94ea9b.slice/crio-aa19af1ec3716d097a5a68083b6a2ca9403e09f19aa01f2c8c5f0103b7620f14 WatchSource:0}: Error finding container aa19af1ec3716d097a5a68083b6a2ca9403e09f19aa01f2c8c5f0103b7620f14: Status 404 returned error can't find the container with id aa19af1ec3716d097a5a68083b6a2ca9403e09f19aa01f2c8c5f0103b7620f14
Oct 31 04:15:01.016263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.016235    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:01.021133 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.021106    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65f9c12_51ba_4ee2_897e_23261a4971aa.slice/crio-859f919b46757f56431045e956202ecc9de021d98b900341f17b34b73abcd501 WatchSource:0}: Error finding container 859f919b46757f56431045e956202ecc9de021d98b900341f17b34b73abcd501: Status 404 returned error can't find the container with id 859f919b46757f56431045e956202ecc9de021d98b900341f17b34b73abcd501
Oct 31 04:15:01.034828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.034802    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:01.037369 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.037339    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c9171_02ca_4982_b261_9434d25f43ac.slice/crio-a7fa83d1200f6b3c6da6924569586319cb112786256d28c2fe53a8fdeba3f0e5 WatchSource:0}: Error finding container a7fa83d1200f6b3c6da6924569586319cb112786256d28c2fe53a8fdeba3f0e5: Status 404 returned error can't find the container with id a7fa83d1200f6b3c6da6924569586319cb112786256d28c2fe53a8fdeba3f0e5
Oct 31 04:15:01.052388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.052307    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:01.057926 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.057822    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47df3bf_72e3_4804_8780_ab1b7dc14bca.slice/crio-7e9b2f771b6c1866b940d967b40fea887d217402f4df373993620e9508fe7a9d WatchSource:0}: Error finding container 7e9b2f771b6c1866b940d967b40fea887d217402f4df373993620e9508fe7a9d: Status 404 returned error can't find the container with id 7e9b2f771b6c1866b940d967b40fea887d217402f4df373993620e9508fe7a9d
Oct 31 04:15:01.058116 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.058089    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:01.058308 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.058287    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dde262_7fd8_490b_b3de_9e9aa3b5989a.slice/crio-9d5103cc184897d4a87ddfba0d6733bb011dc796d1cc2b3fdd68e401a7077428 WatchSource:0}: Error finding container 9d5103cc184897d4a87ddfba0d6733bb011dc796d1cc2b3fdd68e401a7077428: Status 404 returned error can't find the container with id 9d5103cc184897d4a87ddfba0d6733bb011dc796d1cc2b3fdd68e401a7077428
Oct 31 04:15:01.058937 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:01.058880    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2bf015_2f14_4c4c_876e_6d7781f80323.slice/crio-8e80c7c9692d33169f0dbabc17b0e4532a5729648648c74e4f2c514d349027f1 WatchSource:0}: Error finding container 8e80c7c9692d33169f0dbabc17b0e4532a5729648648c74e4f2c514d349027f1: Status 404 returned error can't find the container with id 8e80c7c9692d33169f0dbabc17b0e4532a5729648648c74e4f2c514d349027f1
Oct 31 04:15:01.063092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.063072    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:01.110435 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.110178    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" event={"ID":"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b","Type":"ContainerStarted","Data":"aa19af1ec3716d097a5a68083b6a2ca9403e09f19aa01f2c8c5f0103b7620f14"}
Oct 31 04:15:01.111685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.111449    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerStarted","Data":"859f919b46757f56431045e956202ecc9de021d98b900341f17b34b73abcd501"}
Oct 31 04:15:01.112715 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.112693    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerStarted","Data":"a7fa83d1200f6b3c6da6924569586319cb112786256d28c2fe53a8fdeba3f0e5"}
Oct 31 04:15:01.113980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.113959    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerStarted","Data":"7e9b2f771b6c1866b940d967b40fea887d217402f4df373993620e9508fe7a9d"}
Oct 31 04:15:01.115203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.115179    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerStarted","Data":"9d5103cc184897d4a87ddfba0d6733bb011dc796d1cc2b3fdd68e401a7077428"}
Oct 31 04:15:01.116433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.116409    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerStarted","Data":"bcd719e68dcd6ac6d0ab879c00f00833ddad24a4ca2183eda4d0c1a6a78a00d8"}
Oct 31 04:15:01.117718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:01.117696    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerStarted","Data":"8e80c7c9692d33169f0dbabc17b0e4532a5729648648c74e4f2c514d349027f1"}
Oct 31 04:15:02.139260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.139215    5831 generic.go:356] "Generic (PLEG): container finished" podID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerID="540819f60fdd6ae8f889ac40d0d5b925368e6a245b8d912e3c23e7da9e1a01d5" exitCode=60
Oct 31 04:15:02.139782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.139750    5831 scope.go:122] "RemoveContainer" containerID="540819f60fdd6ae8f889ac40d0d5b925368e6a245b8d912e3c23e7da9e1a01d5"
Oct 31 04:15:02.142866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.142839    5831 generic.go:356] "Generic (PLEG): container finished" podID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerID="f4356763afb0e527c557dad5c93efd1ee885f3612177cd7aabb15dcaa836b391" exitCode=60
Oct 31 04:15:02.143303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.143281    5831 scope.go:122] "RemoveContainer" containerID="f4356763afb0e527c557dad5c93efd1ee885f3612177cd7aabb15dcaa836b391"
Oct 31 04:15:02.147755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.147728    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerStarted","Data":"98e10674dfc30fa1bdc7d11f6ce2d188a800de4e199ed40d77f019e71f5b8f6d"}
Oct 31 04:15:02.147880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.147782    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" event={"ID":"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b","Type":"ContainerDied","Data":"540819f60fdd6ae8f889ac40d0d5b925368e6a245b8d912e3c23e7da9e1a01d5"}
Oct 31 04:15:02.147880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.147808    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerDied","Data":"f4356763afb0e527c557dad5c93efd1ee885f3612177cd7aabb15dcaa836b391"}
Oct 31 04:15:02.147880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.147835    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerStarted","Data":"226acd9b07ced10054137f750071af5778132a0d5fd41d2cfa2276e7014603f6"}
Oct 31 04:15:02.147880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.147856    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerStarted","Data":"9ade5cf8558121414e5e792c9ed4fb81a757be591728d001df0964fedc71a472"}
Oct 31 04:15:02.149146 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.149121    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerStarted","Data":"12c45ca479a2958654755b70c38acf751c541ceb44070b679a9671934527718b"}
Oct 31 04:15:02.157628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.157600    5831 generic.go:356] "Generic (PLEG): container finished" podID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerID="896a3f3a36a750ac91e72de59aab9074d9e37dbbe42bbb4cec859c9b53fd6598" exitCode=60
Oct 31 04:15:02.157752 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.157691    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerDied","Data":"896a3f3a36a750ac91e72de59aab9074d9e37dbbe42bbb4cec859c9b53fd6598"}
Oct 31 04:15:02.158125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.158107    5831 scope.go:122] "RemoveContainer" containerID="896a3f3a36a750ac91e72de59aab9074d9e37dbbe42bbb4cec859c9b53fd6598"
Oct 31 04:15:02.238589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.238439    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" podStartSLOduration=2.238420948 podStartE2EDuration="2.238420948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.237070452 +0000 UTC m=+4052.251632160" watchObservedRunningTime="2026-10-31 04:15:02.238420948 +0000 UTC m=+4052.252982676"
Oct 31 04:15:02.554032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.553978    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" podStartSLOduration=2.553964507 podStartE2EDuration="2.553964507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.550225101 +0000 UTC m=+4052.564786809" watchObservedRunningTime="2026-10-31 04:15:02.553964507 +0000 UTC m=+4052.568526227"
Oct 31 04:15:02.658548 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.658483    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" podStartSLOduration=2.658469665 podStartE2EDuration="2.658469665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.654060856 +0000 UTC m=+4052.668622570" watchObservedRunningTime="2026-10-31 04:15:02.658469665 +0000 UTC m=+4052.673031379"
Oct 31 04:15:02.864407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:02.864297    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" podStartSLOduration=2.864283098 podStartE2EDuration="2.864283098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.862764658 +0000 UTC m=+4052.877326376" watchObservedRunningTime="2026-10-31 04:15:02.864283098 +0000 UTC m=+4052.878844805"
Oct 31 04:15:03.089636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.089610    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:15:03.168034 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.167945    5831 generic.go:356] "Generic (PLEG): container finished" podID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerID="226acd9b07ced10054137f750071af5778132a0d5fd41d2cfa2276e7014603f6" exitCode=60
Oct 31 04:15:03.168485 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.168024    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerDied","Data":"226acd9b07ced10054137f750071af5778132a0d5fd41d2cfa2276e7014603f6"}
Oct 31 04:15:03.168485 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.168364    5831 scope.go:122] "RemoveContainer" containerID="226acd9b07ced10054137f750071af5778132a0d5fd41d2cfa2276e7014603f6"
Oct 31 04:15:03.169988 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.169964    5831 generic.go:356] "Generic (PLEG): container finished" podID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerID="9ade5cf8558121414e5e792c9ed4fb81a757be591728d001df0964fedc71a472" exitCode=60
Oct 31 04:15:03.170141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.170040    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerDied","Data":"9ade5cf8558121414e5e792c9ed4fb81a757be591728d001df0964fedc71a472"}
Oct 31 04:15:03.170294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.170278    5831 scope.go:122] "RemoveContainer" containerID="9ade5cf8558121414e5e792c9ed4fb81a757be591728d001df0964fedc71a472"
Oct 31 04:15:03.172272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.172075    5831 generic.go:356] "Generic (PLEG): container finished" podID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerID="12c45ca479a2958654755b70c38acf751c541ceb44070b679a9671934527718b" exitCode=60
Oct 31 04:15:03.172272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.172190    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerDied","Data":"12c45ca479a2958654755b70c38acf751c541ceb44070b679a9671934527718b"}
Oct 31 04:15:03.172452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.172357    5831 scope.go:122] "RemoveContainer" containerID="12c45ca479a2958654755b70c38acf751c541ceb44070b679a9671934527718b"
Oct 31 04:15:03.174721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.174698    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerStarted","Data":"860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"}
Oct 31 04:15:03.176486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.176466    5831 generic.go:356] "Generic (PLEG): container finished" podID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerID="98e10674dfc30fa1bdc7d11f6ce2d188a800de4e199ed40d77f019e71f5b8f6d" exitCode=60
Oct 31 04:15:03.176619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.176577    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerDied","Data":"98e10674dfc30fa1bdc7d11f6ce2d188a800de4e199ed40d77f019e71f5b8f6d"}
Oct 31 04:15:03.176753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.176740    5831 scope.go:122] "RemoveContainer" containerID="98e10674dfc30fa1bdc7d11f6ce2d188a800de4e199ed40d77f019e71f5b8f6d"
Oct 31 04:15:03.178566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.178509    5831 generic.go:356] "Generic (PLEG): container finished" podID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerID="582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038" exitCode=60
Oct 31 04:15:03.178566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.178558    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" event={"ID":"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b","Type":"ContainerDied","Data":"582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038"}
Oct 31 04:15:03.178733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.178604    5831 scope.go:122] "RemoveContainer" containerID="540819f60fdd6ae8f889ac40d0d5b925368e6a245b8d912e3c23e7da9e1a01d5"
Oct 31 04:15:03.178824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.178807    5831 scope.go:122] "RemoveContainer" containerID="582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038"
Oct 31 04:15:03.179106 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:03.179082    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt_odh-ai-gateway-infra(3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b"
Oct 31 04:15:03.180887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.180863    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerStarted","Data":"5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"}
Oct 31 04:15:03.623290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.623235    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" podStartSLOduration=3.623220402 podStartE2EDuration="3.623220402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.619960985 +0000 UTC m=+4053.634522716" watchObservedRunningTime="2026-10-31 04:15:03.623220402 +0000 UTC m=+4053.637782144"
Oct 31 04:15:03.938752 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:03.938702    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" podStartSLOduration=3.938686203 podStartE2EDuration="3.938686203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:03.934498827 +0000 UTC m=+4053.949060536" watchObservedRunningTime="2026-10-31 04:15:03.938686203 +0000 UTC m=+4053.953247925"
Oct 31 04:15:04.197933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.197897    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerStarted","Data":"c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"}
Oct 31 04:15:04.199874 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.199852    5831 scope.go:122] "RemoveContainer" containerID="582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038"
Oct 31 04:15:04.200159 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:04.200139    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt_odh-ai-gateway-infra(3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b"
Oct 31 04:15:04.201759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.201732    5831 generic.go:356] "Generic (PLEG): container finished" podID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerID="5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2" exitCode=60
Oct 31 04:15:04.201889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.201868    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerDied","Data":"5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"}
Oct 31 04:15:04.201949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.201928    5831 scope.go:122] "RemoveContainer" containerID="f4356763afb0e527c557dad5c93efd1ee885f3612177cd7aabb15dcaa836b391"
Oct 31 04:15:04.202330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.202029    5831 scope.go:122] "RemoveContainer" containerID="5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"
Oct 31 04:15:04.202330 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:04.202282    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn_odh-ai-gateway-infra(f65f9c12-51ba-4ee2-897e-23261a4971aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa"
Oct 31 04:15:04.204134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.204114    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerStarted","Data":"7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"}
Oct 31 04:15:04.206029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.205999    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerStarted","Data":"285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"}
Oct 31 04:15:04.208177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.208152    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerStarted","Data":"57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"}
Oct 31 04:15:04.210393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.210372    5831 generic.go:356] "Generic (PLEG): container finished" podID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerID="860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2" exitCode=60
Oct 31 04:15:04.210510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.210412    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerDied","Data":"860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"}
Oct 31 04:15:04.210627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.210613    5831 scope.go:122] "RemoveContainer" containerID="860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"
Oct 31 04:15:04.210830 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:04.210813    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6_odh-ai-gateway-infra(1e9a34cf-a392-476d-97f4-db31bec14cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6"
Oct 31 04:15:04.217019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:04.216996    5831 scope.go:122] "RemoveContainer" containerID="896a3f3a36a750ac91e72de59aab9074d9e37dbbe42bbb4cec859c9b53fd6598"
Oct 31 04:15:05.221353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.221295    5831 generic.go:356] "Generic (PLEG): container finished" podID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerID="285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125" exitCode=60
Oct 31 04:15:05.221928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.221372    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerDied","Data":"285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"}
Oct 31 04:15:05.221928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.221469    5831 scope.go:122] "RemoveContainer" containerID="9ade5cf8558121414e5e792c9ed4fb81a757be591728d001df0964fedc71a472"
Oct 31 04:15:05.222036 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.221975    5831 scope.go:122] "RemoveContainer" containerID="285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"
Oct 31 04:15:05.222357 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.222327    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl_odh-ai-gateway-infra(e47df3bf-72e3-4804-8780-ab1b7dc14bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca"
Oct 31 04:15:05.224153 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.224128    5831 generic.go:356] "Generic (PLEG): container finished" podID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerID="57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf" exitCode=60
Oct 31 04:15:05.224290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.224209    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerDied","Data":"57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"}
Oct 31 04:15:05.224501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.224485    5831 scope.go:122] "RemoveContainer" containerID="57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"
Oct 31 04:15:05.224866 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.224841    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-g9xtq_odh-ai-gateway-infra(44dde262-7fd8-490b-b3de-9e9aa3b5989a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a"
Oct 31 04:15:05.226628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.226606    5831 scope.go:122] "RemoveContainer" containerID="860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"
Oct 31 04:15:05.226899 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.226865    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6_odh-ai-gateway-infra(1e9a34cf-a392-476d-97f4-db31bec14cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6"
Oct 31 04:15:05.228676 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.228654    5831 generic.go:356] "Generic (PLEG): container finished" podID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerID="c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d" exitCode=60
Oct 31 04:15:05.228787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.228733    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerDied","Data":"c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"}
Oct 31 04:15:05.228941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.228927    5831 scope.go:122] "RemoveContainer" containerID="c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"
Oct 31 04:15:05.229188 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.229169    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx_odh-ai-gateway-infra(da2bf015-2f14-4c4c-876e-6d7781f80323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323"
Oct 31 04:15:05.231018 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.230996    5831 scope.go:122] "RemoveContainer" containerID="5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"
Oct 31 04:15:05.231303 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.231275    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn_odh-ai-gateway-infra(f65f9c12-51ba-4ee2-897e-23261a4971aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa"
Oct 31 04:15:05.232798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.232778    5831 generic.go:356] "Generic (PLEG): container finished" podID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerID="7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995" exitCode=60
Oct 31 04:15:05.232898 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.232857    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerDied","Data":"7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"}
Oct 31 04:15:05.233049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.233032    5831 scope.go:122] "RemoveContainer" containerID="7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"
Oct 31 04:15:05.233280 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:05.233258    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn_odh-ai-gateway-infra(ea2c9171-02ca-4982-b261-9434d25f43ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac"
Oct 31 04:15:05.239071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.239043    5831 scope.go:122] "RemoveContainer" containerID="12c45ca479a2958654755b70c38acf751c541ceb44070b679a9671934527718b"
Oct 31 04:15:05.268193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.268166    5831 scope.go:122] "RemoveContainer" containerID="98e10674dfc30fa1bdc7d11f6ce2d188a800de4e199ed40d77f019e71f5b8f6d"
Oct 31 04:15:05.283363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:05.283338    5831 scope.go:122] "RemoveContainer" containerID="226acd9b07ced10054137f750071af5778132a0d5fd41d2cfa2276e7014603f6"
Oct 31 04:15:06.243313 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:06.243288    5831 scope.go:122] "RemoveContainer" containerID="c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"
Oct 31 04:15:06.243788 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:06.243575    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx_odh-ai-gateway-infra(da2bf015-2f14-4c4c-876e-6d7781f80323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323"
Oct 31 04:15:06.245197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:06.245169    5831 scope.go:122] "RemoveContainer" containerID="7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"
Oct 31 04:15:06.245460 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:06.245422    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn_odh-ai-gateway-infra(ea2c9171-02ca-4982-b261-9434d25f43ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac"
Oct 31 04:15:06.246893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:06.246876    5831 scope.go:122] "RemoveContainer" containerID="285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"
Oct 31 04:15:06.247099 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:06.247078    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl_odh-ai-gateway-infra(e47df3bf-72e3-4804-8780-ab1b7dc14bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca"
Oct 31 04:15:06.248380 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:06.248364    5831 scope.go:122] "RemoveContainer" containerID="57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"
Oct 31 04:15:06.248593 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:06.248575    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-g9xtq_odh-ai-gateway-infra(44dde262-7fd8-490b-b3de-9e9aa3b5989a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a"
Oct 31 04:15:12.749622 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:12.749578    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:15:12.982619 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:12.982579    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:15:13.128078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:13.127994    5831 scope.go:122] "RemoveContainer" containerID="582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038"
Oct 31 04:15:14.128093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.128058    5831 scope.go:122] "RemoveContainer" containerID="5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"
Oct 31 04:15:14.434983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.434939    5831 generic.go:356] "Generic (PLEG): container finished" podID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerID="317dee542c2913c047b24f2c0f919227d67e771cbe901b61c942e7adfa7ef529" exitCode=60
Oct 31 04:15:14.435420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.435041    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" event={"ID":"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b","Type":"ContainerDied","Data":"317dee542c2913c047b24f2c0f919227d67e771cbe901b61c942e7adfa7ef529"}
Oct 31 04:15:14.435420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.435101    5831 scope.go:122] "RemoveContainer" containerID="582aabffcd8c6b09a49e4dd156e38e24ca4e87241593c4c2bd4ac3e35e2b3038"
Oct 31 04:15:14.435669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.435649    5831 scope.go:122] "RemoveContainer" containerID="317dee542c2913c047b24f2c0f919227d67e771cbe901b61c942e7adfa7ef529"
Oct 31 04:15:14.436193 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:14.436032    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt_odh-ai-gateway-infra(3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b"
Oct 31 04:15:14.440286 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:14.440260    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerStarted","Data":"2e48f727463909f7c497a9255cfcc298bee6860316d9b5f71ca4e88c1de1dafe"}
Oct 31 04:15:15.127905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.127867    5831 scope.go:122] "RemoveContainer" containerID="57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"
Oct 31 04:15:15.128076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.127943    5831 scope.go:122] "RemoveContainer" containerID="860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"
Oct 31 04:15:15.451871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.451830    5831 generic.go:356] "Generic (PLEG): container finished" podID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerID="2e48f727463909f7c497a9255cfcc298bee6860316d9b5f71ca4e88c1de1dafe" exitCode=60
Oct 31 04:15:15.452311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.451900    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerDied","Data":"2e48f727463909f7c497a9255cfcc298bee6860316d9b5f71ca4e88c1de1dafe"}
Oct 31 04:15:15.452311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.451961    5831 scope.go:122] "RemoveContainer" containerID="5d1a0c8be1369792a4e7d41885737fc84d2f1ab8c400a21dca568c2bf9f575e2"
Oct 31 04:15:15.452453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.452352    5831 scope.go:122] "RemoveContainer" containerID="2e48f727463909f7c497a9255cfcc298bee6860316d9b5f71ca4e88c1de1dafe"
Oct 31 04:15:15.452727 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:15.452708    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn_odh-ai-gateway-infra(f65f9c12-51ba-4ee2-897e-23261a4971aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa"
Oct 31 04:15:15.566124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.566092    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:15.665283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.665199    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:15.756438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.756414    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:15.836192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.836163    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\") pod \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") "
Oct 31 04:15:15.836389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.836228    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\" (UniqueName: \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\") pod \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\" (UID: \"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b\") "
Oct 31 04:15:15.836648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.836624    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca" pod "3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" (UID: "3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:15.836866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.836849    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:15.838667 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.838645    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp" pod "3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" (UID: "3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b"). InnerVolumeSpecName "kube-api-access-z5sbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:15.938366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:15.938318    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5sbp\" (UniqueName: \"kubernetes.io/projected/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b-kube-api-access-z5sbp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:16.460900 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.460818    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"
Oct 31 04:15:16.461426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.460820    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt" event={"ID":"3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b","Type":"ContainerDied","Data":"aa19af1ec3716d097a5a68083b6a2ca9403e09f19aa01f2c8c5f0103b7620f14"}
Oct 31 04:15:16.461426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.460965    5831 scope.go:122] "RemoveContainer" containerID="317dee542c2913c047b24f2c0f919227d67e771cbe901b61c942e7adfa7ef529"
Oct 31 04:15:16.465021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.464993    5831 generic.go:356] "Generic (PLEG): container finished" podID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerID="876ae6b308f4a903d82026512a7a672b86fe3e120050f3626f0ba22e5a0ecd20" exitCode=60
Oct 31 04:15:16.465126 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.465068    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerDied","Data":"876ae6b308f4a903d82026512a7a672b86fe3e120050f3626f0ba22e5a0ecd20"}
Oct 31 04:15:16.465516 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.465497    5831 scope.go:122] "RemoveContainer" containerID="876ae6b308f4a903d82026512a7a672b86fe3e120050f3626f0ba22e5a0ecd20"
Oct 31 04:15:16.465865 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:16.465841    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-g9xtq_odh-ai-gateway-infra(44dde262-7fd8-490b-b3de-9e9aa3b5989a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a"
Oct 31 04:15:16.467802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.467780    5831 generic.go:356] "Generic (PLEG): container finished" podID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerID="3aa7a9f1e644b384ddc65867239598189622f60c1088b6f769f8a33d18db9bfb" exitCode=60
Oct 31 04:15:16.467912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.467851    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerDied","Data":"3aa7a9f1e644b384ddc65867239598189622f60c1088b6f769f8a33d18db9bfb"}
Oct 31 04:15:16.468264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.468247    5831 scope.go:122] "RemoveContainer" containerID="3aa7a9f1e644b384ddc65867239598189622f60c1088b6f769f8a33d18db9bfb"
Oct 31 04:15:16.468633 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:16.468610    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6_odh-ai-gateway-infra(1e9a34cf-a392-476d-97f4-db31bec14cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6"
Oct 31 04:15:16.473124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.473103    5831 scope.go:122] "RemoveContainer" containerID="57d602d475ef8b40cb7da07f4a020c3629045f50da6a365f40f22e78bcce1ebf"
Oct 31 04:15:16.528600 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.524754    5831 scope.go:122] "RemoveContainer" containerID="860c7c4057ab6275292cfb1214a1f75f0c7c91d33cbcae23857aacdfb23c85a2"
Oct 31 04:15:16.673974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.673951    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:16.747943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.747849    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\" (UniqueName: \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\") pod \"f65f9c12-51ba-4ee2-897e-23261a4971aa\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") "
Oct 31 04:15:16.748178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.748148    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\") pod \"f65f9c12-51ba-4ee2-897e-23261a4971aa\" (UID: \"f65f9c12-51ba-4ee2-897e-23261a4971aa\") "
Oct 31 04:15:16.748495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.748472    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca" pod "f65f9c12-51ba-4ee2-897e-23261a4971aa" (UID: "f65f9c12-51ba-4ee2-897e-23261a4971aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:16.748804 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.748697    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f9c12-51ba-4ee2-897e-23261a4971aa-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:16.750259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.750237    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2" pod "f65f9c12-51ba-4ee2-897e-23261a4971aa" (UID: "f65f9c12-51ba-4ee2-897e-23261a4971aa"). InnerVolumeSpecName "kube-api-access-zs2p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:16.850297 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.850261    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2p2\" (UniqueName: \"kubernetes.io/projected/f65f9c12-51ba-4ee2-897e-23261a4971aa-kube-api-access-zs2p2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:16.945488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.945455    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:16.949357 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:16.949332    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890335-9k4rt"]
Oct 31 04:15:17.517782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.517735    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn" event={"ID":"f65f9c12-51ba-4ee2-897e-23261a4971aa","Type":"ContainerDied","Data":"859f919b46757f56431045e956202ecc9de021d98b900341f17b34b73abcd501"}
Oct 31 04:15:17.518363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.518342    5831 scope.go:122] "RemoveContainer" containerID="2e48f727463909f7c497a9255cfcc298bee6860316d9b5f71ca4e88c1de1dafe"
Oct 31 04:15:17.519132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.518483    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"
Oct 31 04:15:17.647324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.647284    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:17.679783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.679750    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:17.696078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.696043    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d39257-29890335-fpvgn"]
Oct 31 04:15:17.818079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.818043    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:17.852660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.852622    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:17.868176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.868121    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\" (UniqueName: \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\") pod \"1e9a34cf-a392-476d-97f4-db31bec14cb6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") "
Oct 31 04:15:17.868493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.868343    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\") pod \"1e9a34cf-a392-476d-97f4-db31bec14cb6\" (UID: \"1e9a34cf-a392-476d-97f4-db31bec14cb6\") "
Oct 31 04:15:17.869611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.869502    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca" pod "1e9a34cf-a392-476d-97f4-db31bec14cb6" (UID: "1e9a34cf-a392-476d-97f4-db31bec14cb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:17.871942 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.871907    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w" pod "1e9a34cf-a392-476d-97f4-db31bec14cb6" (UID: "1e9a34cf-a392-476d-97f4-db31bec14cb6"). InnerVolumeSpecName "kube-api-access-dn62w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:17.971189 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.971147    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9a34cf-a392-476d-97f4-db31bec14cb6-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:17.971189 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:17.971190    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn62w\" (UniqueName: \"kubernetes.io/projected/1e9a34cf-a392-476d-97f4-db31bec14cb6-kube-api-access-dn62w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.036002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.035936    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:18.072247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.072214    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\") pod \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") "
Oct 31 04:15:18.072442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.072349    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\" (UniqueName: \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\") pod \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\" (UID: \"44dde262-7fd8-490b-b3de-9e9aa3b5989a\") "
Oct 31 04:15:18.072701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.072676    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca" pod "44dde262-7fd8-490b-b3de-9e9aa3b5989a" (UID: "44dde262-7fd8-490b-b3de-9e9aa3b5989a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:18.073074 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.073053    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dde262-7fd8-490b-b3de-9e9aa3b5989a-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.074707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.074683    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx" pod "44dde262-7fd8-490b-b3de-9e9aa3b5989a" (UID: "44dde262-7fd8-490b-b3de-9e9aa3b5989a"). InnerVolumeSpecName "kube-api-access-cxzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:18.129049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.128580    5831 scope.go:122] "RemoveContainer" containerID="7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"
Oct 31 04:15:18.129049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.128731    5831 scope.go:122] "RemoveContainer" containerID="285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"
Oct 31 04:15:18.129049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.128831    5831 scope.go:122] "RemoveContainer" containerID="c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"
Oct 31 04:15:18.140876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.140841    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" path="/var/lib/kubelet/pods/3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b/volumes"
Oct 31 04:15:18.141472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.141448    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" path="/var/lib/kubelet/pods/f65f9c12-51ba-4ee2-897e-23261a4971aa/volumes"
Oct 31 04:15:18.176176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.176145    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzdx\" (UniqueName: \"kubernetes.io/projected/44dde262-7fd8-490b-b3de-9e9aa3b5989a-kube-api-access-cxzdx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.537693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.537664    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"
Oct 31 04:15:18.538180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.537694    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq" event={"ID":"44dde262-7fd8-490b-b3de-9e9aa3b5989a","Type":"ContainerDied","Data":"9d5103cc184897d4a87ddfba0d6733bb011dc796d1cc2b3fdd68e401a7077428"}
Oct 31 04:15:18.538180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.537779    5831 scope.go:122] "RemoveContainer" containerID="876ae6b308f4a903d82026512a7a672b86fe3e120050f3626f0ba22e5a0ecd20"
Oct 31 04:15:18.539497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.539471    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"
Oct 31 04:15:18.539632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.539497    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6" event={"ID":"1e9a34cf-a392-476d-97f4-db31bec14cb6","Type":"ContainerDied","Data":"bcd719e68dcd6ac6d0ab879c00f00833ddad24a4ca2183eda4d0c1a6a78a00d8"}
Oct 31 04:15:18.550068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.550049    5831 scope.go:122] "RemoveContainer" containerID="3aa7a9f1e644b384ddc65867239598189622f60c1088b6f769f8a33d18db9bfb"
Oct 31 04:15:18.691407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.691370    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:18.701797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.701751    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-g9xtq"]
Oct 31 04:15:18.801761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.801732    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:18.809751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:18.809725    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-861dfd-29890335-zffm6"]
Oct 31 04:15:19.552232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.552145    5831 generic.go:356] "Generic (PLEG): container finished" podID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerID="769b09de1cdf0ce79be024e4425887ad02987f113dc1bf85704dfd4d37248435" exitCode=60
Oct 31 04:15:19.552232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.552221    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerDied","Data":"769b09de1cdf0ce79be024e4425887ad02987f113dc1bf85704dfd4d37248435"}
Oct 31 04:15:19.552791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.552284    5831 scope.go:122] "RemoveContainer" containerID="7783a08fb36fe71fd605583f0a6b65dd9801b7aab0dcdc66b67fdf1fa7bc3995"
Oct 31 04:15:19.552791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.552753    5831 scope.go:122] "RemoveContainer" containerID="769b09de1cdf0ce79be024e4425887ad02987f113dc1bf85704dfd4d37248435"
Oct 31 04:15:19.553111 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:19.553085    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn_odh-ai-gateway-infra(ea2c9171-02ca-4982-b261-9434d25f43ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac"
Oct 31 04:15:19.554680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.554659    5831 generic.go:356] "Generic (PLEG): container finished" podID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerID="69d6dde1dc2e57d46ec7d80bcdb594df8f34ab54eff4774aff1948abb6bed8f9" exitCode=60
Oct 31 04:15:19.554778 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.554734    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerDied","Data":"69d6dde1dc2e57d46ec7d80bcdb594df8f34ab54eff4774aff1948abb6bed8f9"}
Oct 31 04:15:19.555160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.555146    5831 scope.go:122] "RemoveContainer" containerID="69d6dde1dc2e57d46ec7d80bcdb594df8f34ab54eff4774aff1948abb6bed8f9"
Oct 31 04:15:19.555464 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:19.555444    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl_odh-ai-gateway-infra(e47df3bf-72e3-4804-8780-ab1b7dc14bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca"
Oct 31 04:15:19.559508 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.559483    5831 generic.go:356] "Generic (PLEG): container finished" podID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerID="042339f2530a5e6b6c3ebcdbc864f5a46dc253654e8fe39312397aa75ac600c9" exitCode=60
Oct 31 04:15:19.559628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.559560    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerDied","Data":"042339f2530a5e6b6c3ebcdbc864f5a46dc253654e8fe39312397aa75ac600c9"}
Oct 31 04:15:19.559931 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.559916    5831 scope.go:122] "RemoveContainer" containerID="042339f2530a5e6b6c3ebcdbc864f5a46dc253654e8fe39312397aa75ac600c9"
Oct 31 04:15:19.560185 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:19.560169    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx_odh-ai-gateway-infra(da2bf015-2f14-4c4c-876e-6d7781f80323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323"
Oct 31 04:15:19.567381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.567362    5831 scope.go:122] "RemoveContainer" containerID="285e4868bff30a79727f7c52d902473c9290a83d0407a02706a21ec9c4b2d125"
Oct 31 04:15:19.585820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:19.584053    5831 scope.go:122] "RemoveContainer" containerID="c09f83dd4dae963ebe8c6706c4923ea72dd11f5ef268bea0cc02e9c14aa3393d"
Oct 31 04:15:20.138621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.138592    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" path="/var/lib/kubelet/pods/1e9a34cf-a392-476d-97f4-db31bec14cb6/volumes"
Oct 31 04:15:20.139124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.139103    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" path="/var/lib/kubelet/pods/44dde262-7fd8-490b-b3de-9e9aa3b5989a/volumes"
Oct 31 04:15:20.673889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.671187    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:20.798578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.784084    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:20.894736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.894693    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:20.897647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.897603    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:20.991599 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:20.989316    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:21.052674 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.052635    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\" (UniqueName: \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\") pod \"ea2c9171-02ca-4982-b261-9434d25f43ac\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") "
Oct 31 04:15:21.052850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.052821    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\") pod \"ea2c9171-02ca-4982-b261-9434d25f43ac\" (UID: \"ea2c9171-02ca-4982-b261-9434d25f43ac\") "
Oct 31 04:15:21.053299 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.053268    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca" pod "ea2c9171-02ca-4982-b261-9434d25f43ac" (UID: "ea2c9171-02ca-4982-b261-9434d25f43ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:21.053832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.053795    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c9171-02ca-4982-b261-9434d25f43ac-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.063232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.063201    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7" pod "ea2c9171-02ca-4982-b261-9434d25f43ac" (UID: "ea2c9171-02ca-4982-b261-9434d25f43ac"). InnerVolumeSpecName "kube-api-access-5xmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:21.119086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.119053    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:21.154733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.154697    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\" (UniqueName: \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\") pod \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") "
Oct 31 04:15:21.154947 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.154933    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\") pod \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\" (UID: \"e47df3bf-72e3-4804-8780-ab1b7dc14bca\") "
Oct 31 04:15:21.155375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.155350    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca" pod "e47df3bf-72e3-4804-8780-ab1b7dc14bca" (UID: "e47df3bf-72e3-4804-8780-ab1b7dc14bca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:21.155473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.155465    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmn7\" (UniqueName: \"kubernetes.io/projected/ea2c9171-02ca-4982-b261-9434d25f43ac-kube-api-access-5xmn7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.155518 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.155479    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47df3bf-72e3-4804-8780-ab1b7dc14bca-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.157116 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.157092    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2" pod "e47df3bf-72e3-4804-8780-ab1b7dc14bca" (UID: "e47df3bf-72e3-4804-8780-ab1b7dc14bca"). InnerVolumeSpecName "kube-api-access-bq6l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:21.256391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.256309    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\") pod \"da2bf015-2f14-4c4c-876e-6d7781f80323\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") "
Oct 31 04:15:21.256391 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.256364    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\" (UniqueName: \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\") pod \"da2bf015-2f14-4c4c-876e-6d7781f80323\" (UID: \"da2bf015-2f14-4c4c-876e-6d7781f80323\") "
Oct 31 04:15:21.256748 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.256725    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca" pod "da2bf015-2f14-4c4c-876e-6d7781f80323" (UID: "da2bf015-2f14-4c4c-876e-6d7781f80323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:21.257120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.257094    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bf015-2f14-4c4c-876e-6d7781f80323-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.257120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.257117    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6l2\" (UniqueName: \"kubernetes.io/projected/e47df3bf-72e3-4804-8780-ab1b7dc14bca-kube-api-access-bq6l2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.258991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.258968    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb" pod "da2bf015-2f14-4c4c-876e-6d7781f80323" (UID: "da2bf015-2f14-4c4c-876e-6d7781f80323"). InnerVolumeSpecName "kube-api-access-qbvmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:21.358573 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.358513    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvmb\" (UniqueName: \"kubernetes.io/projected/da2bf015-2f14-4c4c-876e-6d7781f80323-kube-api-access-qbvmb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.582738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.582655    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"
Oct 31 04:15:21.582738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.582667    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx" event={"ID":"da2bf015-2f14-4c4c-876e-6d7781f80323","Type":"ContainerDied","Data":"8e80c7c9692d33169f0dbabc17b0e4532a5729648648c74e4f2c514d349027f1"}
Oct 31 04:15:21.582738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.582732    5831 scope.go:122] "RemoveContainer" containerID="042339f2530a5e6b6c3ebcdbc864f5a46dc253654e8fe39312397aa75ac600c9"
Oct 31 04:15:21.584621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.584602    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"
Oct 31 04:15:21.584772 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.584635    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn" event={"ID":"ea2c9171-02ca-4982-b261-9434d25f43ac","Type":"ContainerDied","Data":"a7fa83d1200f6b3c6da6924569586319cb112786256d28c2fe53a8fdeba3f0e5"}
Oct 31 04:15:21.586476 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.586448    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl" event={"ID":"e47df3bf-72e3-4804-8780-ab1b7dc14bca","Type":"ContainerDied","Data":"7e9b2f771b6c1866b940d967b40fea887d217402f4df373993620e9508fe7a9d"}
Oct 31 04:15:21.586635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.586460    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"
Oct 31 04:15:21.597646 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.597622    5831 scope.go:122] "RemoveContainer" containerID="769b09de1cdf0ce79be024e4425887ad02987f113dc1bf85704dfd4d37248435"
Oct 31 04:15:21.609406 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.609376    5831 scope.go:122] "RemoveContainer" containerID="69d6dde1dc2e57d46ec7d80bcdb594df8f34ab54eff4774aff1948abb6bed8f9"
Oct 31 04:15:21.740078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.740021    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:21.745763 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.745732    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-83f052-29890335-gvksx"]
Oct 31 04:15:21.842342 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.842216    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:21.846943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.846912    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e2a54d-29890335-h67sl"]
Oct 31 04:15:21.946916 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.946883    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:21.950356 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:21.950333    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2339ed-29890335-z2wqn"]
Oct 31 04:15:22.137489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:22.137409    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" path="/var/lib/kubelet/pods/da2bf015-2f14-4c4c-876e-6d7781f80323/volumes"
Oct 31 04:15:22.137932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:22.137917    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" path="/var/lib/kubelet/pods/e47df3bf-72e3-4804-8780-ab1b7dc14bca/volumes"
Oct 31 04:15:22.138352 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:22.138340    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" path="/var/lib/kubelet/pods/ea2c9171-02ca-4982-b261-9434d25f43ac/volumes"
Oct 31 04:15:36.251001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:36.250967    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:15:36.251413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:36.251193    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy" containerID="cri-o://127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae" gracePeriod=30
Oct 31 04:15:41.591519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.591488    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:15:41.711749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711709    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711769    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711798    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711820    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711854    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711892    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.711938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711928    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.712296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.711973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.712296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712049    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\") pod \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\" (UID: \"663a1ddc-8f87-4209-a0e9-a6cd3bb72234\") "
Oct 31 04:15:41.712296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712071    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:15:41.712296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712130    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:15:41.712296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712142    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:15:41.712612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712359    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:41.712825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712730    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.712825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712755    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.712825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712769    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.712825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712782    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.712825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.712785    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:15:41.714752 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.714726    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:15:41.714976 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.714949    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:41.715058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.714973    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "kube-api-access-s8zpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:41.715178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.715161    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo" pod "663a1ddc-8f87-4209-a0e9-a6cd3bb72234" (UID: "663a1ddc-8f87-4209-a0e9-a6cd3bb72234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:15:41.783402 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.783360    5831 generic.go:356] "Generic (PLEG): container finished" podID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerID="127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae" exitCode=0
Oct 31 04:15:41.783654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.783441    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"
Oct 31 04:15:41.783654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.783440    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerDied","Data":"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"}
Oct 31 04:15:41.783654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.783594    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94" event={"ID":"663a1ddc-8f87-4209-a0e9-a6cd3bb72234","Type":"ContainerDied","Data":"13d5d8fddba8546a1ae264bd1a24069e1e7cbfd51ae17cee1fc140f35da11120"}
Oct 31 04:15:41.783654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.783628    5831 scope.go:122] "RemoveContainer" containerID="127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"
Oct 31 04:15:41.799475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.799451    5831 scope.go:122] "RemoveContainer" containerID="d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"
Oct 31 04:15:41.811457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.811437    5831 scope.go:122] "RemoveContainer" containerID="127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"
Oct 31 04:15:41.811837 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:41.811804    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae\": container with ID starting with 127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae not found: ID does not exist" containerID="127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"
Oct 31 04:15:41.811951 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.811840    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae"} err="failed to get container status \"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae\": rpc error: code = NotFound desc = could not find container \"127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae\": container with ID starting with 127a5c67807dd4b3c73a8605fcfece5bd27af9c6c0d0224cfe78ffa4ec86a2ae not found: ID does not exist"
Oct 31 04:15:41.811951 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.811866    5831 scope.go:122] "RemoveContainer" containerID="d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"
Oct 31 04:15:41.812170 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:41.812151    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480\": container with ID starting with d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480 not found: ID does not exist" containerID="d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"
Oct 31 04:15:41.812255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.812182    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480"} err="failed to get container status \"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480\": rpc error: code = NotFound desc = could not find container \"d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480\": container with ID starting with d802f3d1f601f67dfb188d47f37c9d8aeffa04efa63889150414b4c098013480 not found: ID does not exist"
Oct 31 04:15:41.814010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.813986    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.814010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.814014    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zpp\" (UniqueName: \"kubernetes.io/projected/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-kube-api-access-s8zpp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.814185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.814029    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.814185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.814038    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.814185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.814097    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663a1ddc-8f87-4209-a0e9-a6cd3bb72234-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:41.938785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.938742    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:15:41.942980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:41.942954    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d247af-openshift-default-866457bbcc-8bw94"]
Oct 31 04:15:42.136309 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:42.136225    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" path="/var/lib/kubelet/pods/663a1ddc-8f87-4209-a0e9-a6cd3bb72234/volumes"
Oct 31 04:15:42.983515 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:15:42.983480    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:15:48.047551 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.017673    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:15:48.049691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.049660    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.049947 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.049931    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.050078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050066    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.050174 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050160    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.050258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050248    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.050344 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050334    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.050438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050428    5831 cpu_manager.go:414] "containerMap: removing container" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.050522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050512    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.050694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050681    5831 cpu_manager.go:414] "containerMap: removing container" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.050780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050770    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.050866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050856    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.050951 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.050941    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.051038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051028    5831 cpu_manager.go:414] "containerMap: removing container" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.051118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051107    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.051203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051193    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.051283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051273    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.051386 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051376    5831 cpu_manager.go:414] "containerMap: removing container" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.051556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051521    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.051670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051660    5831 cpu_manager.go:414] "containerMap: removing container" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.051754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051744    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.051854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.051844    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.055647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.055623    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.055785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.055773    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.055880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.055869    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.055975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.055964    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.056049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056040    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.056124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056115    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.056195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056185    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.056289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056280    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.056368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056358    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.056454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056445    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.056547 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056519    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.056651 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056639    5831 cpu_manager.go:414] "containerMap: removing container" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.056724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.056715    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.057875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.057845    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.058011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.057998    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.058117 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058107    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.058197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058188    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.058287 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058278    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.058414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058402    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.058757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058741    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.058876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058865    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.058963 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.058954    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.059046 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059037    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.059127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059118    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.059201 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059193    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.059269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059261    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.059349 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059341    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.059425 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059417    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.059495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059487    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.059603 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059579    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.059704 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059694    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.059795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059786    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.059872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059864    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.059988 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.059978    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.200774 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.200740    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:15:48.201502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.201482    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.238130 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238099    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:15:48.238871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238850    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.239006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238871    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.239006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238914    5831 cpu_manager.go:414] "containerMap: removing container" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.239006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238921    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.239006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238934    5831 cpu_manager.go:414] "containerMap: removing container" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.239006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.238941    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239144    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bf015-2f14-4c4c-876e-6d7781f80323" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239166    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9f4d36-2477-4f1b-b40f-8e46fe94ea9b" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239177    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65f9c12-51ba-4ee2-897e-23261a4971aa" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239191    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47df3bf-72e3-4804-8780-ab1b7dc14bca" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239201    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c9171-02ca-4982-b261-9434d25f43ac" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239212    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dde262-7fd8-490b-b3de-9e9aa3b5989a" containerName="cleanup"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239223    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="663a1ddc-8f87-4209-a0e9-a6cd3bb72234" containerName="istio-proxy"
Oct 31 04:15:48.239266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.239233    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9a34cf-a392-476d-97f4-db31bec14cb6" containerName="cleanup"
Oct 31 04:15:48.243134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.243114    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.298675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.298572    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c42c43-dockercfg-ngqqm\""
Oct 31 04:15:48.298675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.298571    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c42c43\""
Oct 31 04:15:48.302792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.302764    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:15:48.327613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.327575    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.327782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.327740    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.327848 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.327819    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7mb\" (UniqueName: \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.327995 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.327967    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kbx\" (UniqueName: \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.428882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.428840    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.429106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.428943    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7mb\" (UniqueName: \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.429106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.429012    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kbx\" (UniqueName: \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.429230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.429106    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.429700 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.429679    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.430003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.429982    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.482196 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.482159    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kbx\" (UniqueName: \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\") pod \"payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.482351 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.482159    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7mb\" (UniqueName: \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\") pod \"payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.540254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.540221    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:15:48.558983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.558958    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:48.928301 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.928272    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:15:48.966523 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:48.966491    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aade3b_cbb1_4259_836c_e03188900fac.slice/crio-04304b79d9dc9742649361f311bff6baba6aa7fa5403e762d5cbbbe84cd0b0fd WatchSource:0}: Error finding container 04304b79d9dc9742649361f311bff6baba6aa7fa5403e762d5cbbbe84cd0b0fd: Status 404 returned error can't find the container with id 04304b79d9dc9742649361f311bff6baba6aa7fa5403e762d5cbbbe84cd0b0fd
Oct 31 04:15:48.976857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:48.976833    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:15:48.980386 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:15:48.980358    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73ed55d_4659_4fd9_af10_7c7dbf189467.slice/crio-08e1ac2973fe9f38396fa51a2ad77db2db5846d1c53cccb288ac9533aadedea9 WatchSource:0}: Error finding container 08e1ac2973fe9f38396fa51a2ad77db2db5846d1c53cccb288ac9533aadedea9: Status 404 returned error can't find the container with id 08e1ac2973fe9f38396fa51a2ad77db2db5846d1c53cccb288ac9533aadedea9
Oct 31 04:15:49.924849 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:49.924660    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" event={"ID":"c4aade3b-cbb1-4259-836c-e03188900fac","Type":"ContainerStarted","Data":"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"}
Oct 31 04:15:49.924849 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:49.924728    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" event={"ID":"c4aade3b-cbb1-4259-836c-e03188900fac","Type":"ContainerStarted","Data":"04304b79d9dc9742649361f311bff6baba6aa7fa5403e762d5cbbbe84cd0b0fd"}
Oct 31 04:15:49.929932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:49.929885    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" event={"ID":"b73ed55d-4659-4fd9-af10-7c7dbf189467","Type":"ContainerStarted","Data":"0f0ef20d051704b6853cc7d82e27b554ba7589fb84ca9692e0e1bdec5cb2dfad"}
Oct 31 04:15:49.931452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:49.931419    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:15:49.931698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:49.931680    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" event={"ID":"b73ed55d-4659-4fd9-af10-7c7dbf189467","Type":"ContainerStarted","Data":"08e1ac2973fe9f38396fa51a2ad77db2db5846d1c53cccb288ac9533aadedea9"}
Oct 31 04:15:50.033393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:50.033324    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" podStartSLOduration=3.033304211 podStartE2EDuration="3.033304211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:50.032748265 +0000 UTC m=+4100.047309972" watchObservedRunningTime="2026-10-31 04:15:50.033304211 +0000 UTC m=+4100.047865920"
Oct 31 04:15:50.145865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:50.145784    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" podStartSLOduration=2.145760728 podStartE2EDuration="2.145760728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:50.137027806 +0000 UTC m=+4100.151589515" watchObservedRunningTime="2026-10-31 04:15:50.145760728 +0000 UTC m=+4100.160322433"
Oct 31 04:15:50.237554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:50.234792    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:15:50.938000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:15:50.937959    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:16:00.110161 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.110119    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"]
Oct 31 04:16:00.115726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.115699    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.165988 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.165951    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"]
Oct 31 04:16:00.206698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.206665    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:16:00.206948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.206743    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:16:00.242614 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.242566    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.242860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.242701    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdl8d\" (UniqueName: \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.242959 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.242940    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.344574 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.344508    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.344798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.344610    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdl8d\" (UniqueName: \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.344798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.344760    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.345194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.345167    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.347284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.347261    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.397437 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.397358    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdl8d\" (UniqueName: \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\") pod \"maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.431655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.431625    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:00.813281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:00.813223    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"]
Oct 31 04:16:00.814142 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:16:00.814112    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa32a48f_34c7_4ff9_8277_fc4181a231cc.slice/crio-1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f WatchSource:0}: Error finding container 1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f: Status 404 returned error can't find the container with id 1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f
Oct 31 04:16:01.031886 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:01.031844    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7" event={"ID":"fa32a48f-34c7-4ff9-8277-fc4181a231cc","Type":"ContainerStarted","Data":"1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f"}
Oct 31 04:16:01.949462 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:01.949433    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:16:01.949903 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:01.949502    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:16:02.040853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:02.040815    5831 generic.go:356] "Generic (PLEG): container finished" podID="fa32a48f-34c7-4ff9-8277-fc4181a231cc" containerID="94bd4e1ccc29d56f3a2fe1607e4ef916bbe333a9ceabdc4e4e8fb74ffa8d8c6d" exitCode=0
Oct 31 04:16:02.041035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:02.040901    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7" event={"ID":"fa32a48f-34c7-4ff9-8277-fc4181a231cc","Type":"ContainerDied","Data":"94bd4e1ccc29d56f3a2fe1607e4ef916bbe333a9ceabdc4e4e8fb74ffa8d8c6d"}
Oct 31 04:16:03.227592 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.227568    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:03.290300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.290267    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\" (UniqueName: \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\") pod \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") "
Oct 31 04:16:03.290509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.290355    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\") pod \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") "
Oct 31 04:16:03.290509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.290420    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\") pod \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\" (UID: \"fa32a48f-34c7-4ff9-8277-fc4181a231cc\") "
Oct 31 04:16:03.290844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.290811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca" pod "fa32a48f-34c7-4ff9-8277-fc4181a231cc" (UID: "fa32a48f-34c7-4ff9-8277-fc4181a231cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:03.291026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.291005    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:03.293340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.293309    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca" pod "fa32a48f-34c7-4ff9-8277-fc4181a231cc" (UID: "fa32a48f-34c7-4ff9-8277-fc4181a231cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:16:03.293427 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.293396    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d" pod "fa32a48f-34c7-4ff9-8277-fc4181a231cc" (UID: "fa32a48f-34c7-4ff9-8277-fc4181a231cc"). InnerVolumeSpecName "kube-api-access-rdl8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:03.393237 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.393151    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdl8d\" (UniqueName: \"kubernetes.io/projected/fa32a48f-34c7-4ff9-8277-fc4181a231cc-kube-api-access-rdl8d\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:03.393237 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:03.393187    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa32a48f-34c7-4ff9-8277-fc4181a231cc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:04.066115 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:04.066079    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"
Oct 31 04:16:04.066300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:04.066080    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7" event={"ID":"fa32a48f-34c7-4ff9-8277-fc4181a231cc","Type":"ContainerDied","Data":"1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f"}
Oct 31 04:16:04.066300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:04.066202    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db710849e27a2f3d6125572d52836c91190defa97e091e81222c5a1ad01654f"
Oct 31 04:16:12.980777 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:16:12.980738    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:16:16.123695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:16.123652    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:16:16.124728 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:16.123880    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy" containerID="cri-o://5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a" gracePeriod=30
Oct 31 04:16:21.421962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.421926    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:16:21.542623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542586    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.542843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542636    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.542843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542662    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.542843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542762    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.542843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542838    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.543075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542892    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.543075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542929    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.543075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.542966    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.543075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543014    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.543300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543093    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\") pod \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\" (UID: \"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc\") "
Oct 31 04:16:21.543300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543233    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.543300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543252    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.543457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543326    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.543761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543737    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.543892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543764    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.543892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543779    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.543892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543793    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.544018 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.543900    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:21.545545 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.545507    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "kube-api-access-7grwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.545717 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.545697    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:21.545809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.545752    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:21.545809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.545784    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token" pod "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" (UID: "94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:21.646384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.645857    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.646384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.645892    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.646384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.645908    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.646384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.645922    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grwj\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-kube-api-access-7grwj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:21.646384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:21.645937    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:22.237133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.237102    5831 generic.go:356] "Generic (PLEG): container finished" podID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerID="5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a" exitCode=0
Oct 31 04:16:22.237329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.237155    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"
Oct 31 04:16:22.237329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.237194    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" event={"ID":"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc","Type":"ContainerDied","Data":"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"}
Oct 31 04:16:22.237329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.237251    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc" event={"ID":"94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc","Type":"ContainerDied","Data":"498f721701756b5ed559e0eef9b037bfb2630702fb97ee1aa6d05cd73fab5f5a"}
Oct 31 04:16:22.237329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.237277    5831 scope.go:122] "RemoveContainer" containerID="5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"
Oct 31 04:16:22.248729 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.248713    5831 scope.go:122] "RemoveContainer" containerID="5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"
Oct 31 04:16:22.249020 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:16:22.249003    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a\": container with ID starting with 5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a not found: ID does not exist" containerID="5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"
Oct 31 04:16:22.249096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.249025    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a"} err="failed to get container status \"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a\": rpc error: code = NotFound desc = could not find container \"5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a\": container with ID starting with 5d6191d8bff85bd2eaf264b6c0006d47b5125df66ff296b41c1502160112e46a not found: ID does not exist"
Oct 31 04:16:22.436592 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.436557    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:16:22.482279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:22.482248    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cb4gc"]
Oct 31 04:16:24.137372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:24.137339    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" path="/var/lib/kubelet/pods/94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc/volumes"
Oct 31 04:16:42.980257 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:16:42.980217    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:16:44.166387 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:44.166295    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:16:49.166647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.166572    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" podUID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" containerName="payload-processing" containerID="cri-o://c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b" gracePeriod=25
Oct 31 04:16:49.441322 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.441290    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:16:49.449449 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.449419    5831 generic.go:356] "Generic (PLEG): container finished" podID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" containerID="c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b" exitCode=0
Oct 31 04:16:49.449607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.449457    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"
Oct 31 04:16:49.449607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.449503    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" event={"ID":"12d1c8c9-f286-4cd5-8863-fa926efb7d65","Type":"ContainerDied","Data":"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"}
Oct 31 04:16:49.449607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.449579    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv" event={"ID":"12d1c8c9-f286-4cd5-8863-fa926efb7d65","Type":"ContainerDied","Data":"13f06aa32ae5fc915b239d9b15384176548bf006a5ed56b0665a455131976ef9"}
Oct 31 04:16:49.449607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.449603    5831 scope.go:122] "RemoveContainer" containerID="c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"
Oct 31 04:16:49.460780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.460754    5831 scope.go:122] "RemoveContainer" containerID="c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"
Oct 31 04:16:49.461066 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:16:49.461044    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b\": container with ID starting with c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b not found: ID does not exist" containerID="c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"
Oct 31 04:16:49.461139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.461082    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b"} err="failed to get container status \"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b\": rpc error: code = NotFound desc = could not find container \"c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b\": container with ID starting with c4921d1b96684261a668e3522ce9985161f718cb2cc544f3bae189a56a16124b not found: ID does not exist"
Oct 31 04:16:49.586850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.586813    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\" (UniqueName: \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\") pod \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") "
Oct 31 04:16:49.587157 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.587122    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\") pod \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\" (UID: \"12d1c8c9-f286-4cd5-8863-fa926efb7d65\") "
Oct 31 04:16:49.587494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.587473    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume" pod "12d1c8c9-f286-4cd5-8863-fa926efb7d65" (UID: "12d1c8c9-f286-4cd5-8863-fa926efb7d65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:49.587710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.587693    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d1c8c9-f286-4cd5-8863-fa926efb7d65-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:49.589874 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.589838    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz" pod "12d1c8c9-f286-4cd5-8863-fa926efb7d65" (UID: "12d1c8c9-f286-4cd5-8863-fa926efb7d65"). InnerVolumeSpecName "kube-api-access-p66wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:49.689193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.689152    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66wz\" (UniqueName: \"kubernetes.io/projected/12d1c8c9-f286-4cd5-8863-fa926efb7d65-kube-api-access-p66wz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:49.862435 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.862370    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:16:49.867740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:49.867713    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e53098-5779d5c545-x4rhv"]
Oct 31 04:16:50.136118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:16:50.136039    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" path="/var/lib/kubelet/pods/12d1c8c9-f286-4cd5-8863-fa926efb7d65/volumes"
Oct 31 04:17:11.905878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:11.905838    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:17:11.906376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:11.906108    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" containerID="cri-o://13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523" gracePeriod=30
Oct 31 04:17:12.489051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:12.489015    5831 patch_prober.go:29] interesting pod/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:17:12.489267 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:12.489090    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:17:12.980134 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:12.980096    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:17:17.200063 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.200025    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:17:17.259736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.259705    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.259957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.259757    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.259957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.259839    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260080 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260000    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260080 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260030    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260080 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260036    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:17.260080 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260072    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260121    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260214    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260258    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\") pod \"b9745416-29f8-4484-9cad-b205d8ecbae1\" (UID: \"b9745416-29f8-4484-9cad-b205d8ecbae1\") "
Oct 31 04:17:17.260303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260260    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:17.260303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260272    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:17.260702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260466    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:17.260702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.260619    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:17.261114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.261089    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9745416-29f8-4484-9cad-b205d8ecbae1-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.261271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.261249    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.261271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.261272    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.261669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.261282    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.261669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.261296    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.262753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.262731    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:17.263210 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.263191    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:17:17.263374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.263352    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:17.263444 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.263388    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv" pod "b9745416-29f8-4484-9cad-b205d8ecbae1" (UID: "b9745416-29f8-4484-9cad-b205d8ecbae1"). InnerVolumeSpecName "kube-api-access-blhzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:17.363104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.363072    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhzv\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-kube-api-access-blhzv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.363104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.363103    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.363104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.363113    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.363353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.363122    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9745416-29f8-4484-9cad-b205d8ecbae1-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:17.666447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.666411    5831 generic.go:356] "Generic (PLEG): container finished" podID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerID="13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523" exitCode=0
Oct 31 04:17:17.666677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.666467    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"
Oct 31 04:17:17.666677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.666500    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerDied","Data":"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"}
Oct 31 04:17:17.666677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.666597    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd" event={"ID":"b9745416-29f8-4484-9cad-b205d8ecbae1","Type":"ContainerDied","Data":"9399fbc53eb5939ac2230ef873aa491721f34ea59519e49e148dd4e2736d10fd"}
Oct 31 04:17:17.666677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.666625    5831 scope.go:122] "RemoveContainer" containerID="13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"
Oct 31 04:17:17.679427 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.679405    5831 scope.go:122] "RemoveContainer" containerID="e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"
Oct 31 04:17:17.693197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.693176    5831 scope.go:122] "RemoveContainer" containerID="13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"
Oct 31 04:17:17.693580 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:17.693554    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523\": container with ID starting with 13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523 not found: ID does not exist" containerID="13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"
Oct 31 04:17:17.693662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.693593    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523"} err="failed to get container status \"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523\": rpc error: code = NotFound desc = could not find container \"13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523\": container with ID starting with 13f64cf791b4c2d7087acc444ea52a12c483980e7114ab7bda76d0656d3c7523 not found: ID does not exist"
Oct 31 04:17:17.693662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.693619    5831 scope.go:122] "RemoveContainer" containerID="e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"
Oct 31 04:17:17.693956 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:17.693930    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe\": container with ID starting with e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe not found: ID does not exist" containerID="e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"
Oct 31 04:17:17.694027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.693965    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe"} err="failed to get container status \"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe\": rpc error: code = NotFound desc = could not find container \"e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe\": container with ID starting with e49eedc805d54beae2a98604fc765973bb3f4cdef6dba32bfbad70283746bebe not found: ID does not exist"
Oct 31 04:17:17.820747 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.820704    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:17:17.866397 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:17.866365    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f0e007-openshift-default-79fbbff9bc-6cbkd"]
Oct 31 04:17:18.138278 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:18.138241    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" path="/var/lib/kubelet/pods/b9745416-29f8-4484-9cad-b205d8ecbae1/volumes"
Oct 31 04:17:26.356899 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:26.356855    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:17:32.822890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:32.822850    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"]
Oct 31 04:17:32.870786 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:32.870746    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e2a54d-bkzx7"]
Oct 31 04:17:33.009698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:33.009657    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:17:33.010006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:33.009959    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" containerID="cri-o://7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7" gracePeriod=30
Oct 31 04:17:34.136932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:34.136893    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa32a48f-34c7-4ff9-8277-fc4181a231cc" path="/var/lib/kubelet/pods/fa32a48f-34c7-4ff9-8277-fc4181a231cc/volumes"
Oct 31 04:17:35.638684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:35.638649    5831 patch_prober.go:29] interesting pod/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:17:35.639115 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:35.638700    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:17:38.319285 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.319257    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:17:38.430512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430479    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430574    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430667    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430692    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430765    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430843    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430835    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:38.430946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430881    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.430946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430927    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.431238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.430978    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\") pod \"784e2fcf-7f87-4d52-9569-92918e2caf8d\" (UID: \"784e2fcf-7f87-4d52-9569-92918e2caf8d\") "
Oct 31 04:17:38.431238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431017    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:38.431238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431036    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:38.431238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431202    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:38.431477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431315    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:38.431619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431585    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.431737 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431676    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784e2fcf-7f87-4d52-9569-92918e2caf8d-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.431792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431780    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.431834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431798    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.431834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.431813    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.433441 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.433399    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "kube-api-access-nlwgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:38.433638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.433609    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:17:38.433905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.433886    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:38.434013 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.433964    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy" pod "784e2fcf-7f87-4d52-9569-92918e2caf8d" (UID: "784e2fcf-7f87-4d52-9569-92918e2caf8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:38.533076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.532992    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.533076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.533019    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.533076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.533031    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwgq\" (UniqueName: \"kubernetes.io/projected/784e2fcf-7f87-4d52-9569-92918e2caf8d-kube-api-access-nlwgq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.533076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.533043    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784e2fcf-7f87-4d52-9569-92918e2caf8d-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:38.609523 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.609496    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:17:38.614329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.614301    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:17:38.621028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.621004    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:17:38.624924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.624904    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:17:38.625208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.625185    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:17:38.629192 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.629092    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:17:38.887955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.887875    5831 generic.go:356] "Generic (PLEG): container finished" podID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerID="7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7" exitCode=0
Oct 31 04:17:38.887955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.887936    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"
Oct 31 04:17:38.888171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.887962    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerDied","Data":"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"}
Oct 31 04:17:38.888171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.888023    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n" event={"ID":"784e2fcf-7f87-4d52-9569-92918e2caf8d","Type":"ContainerDied","Data":"395f2cfcdf2a49bf561f2f6a4b11978bf55f34380e49bac2589a5d42144ebd0d"}
Oct 31 04:17:38.888171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.888056    5831 scope.go:122] "RemoveContainer" containerID="7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"
Oct 31 04:17:38.900984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.900964    5831 scope.go:122] "RemoveContainer" containerID="07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"
Oct 31 04:17:38.912515 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.912494    5831 scope.go:122] "RemoveContainer" containerID="7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"
Oct 31 04:17:38.912892 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:38.912864    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7\": container with ID starting with 7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7 not found: ID does not exist" containerID="7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"
Oct 31 04:17:38.912945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.912902    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7"} err="failed to get container status \"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7\": rpc error: code = NotFound desc = could not find container \"7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7\": container with ID starting with 7edbf2f661d57f262b9f2204fd5610e7c4f44fc23b2f059d911b768cd354d0c7 not found: ID does not exist"
Oct 31 04:17:38.912945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.912921    5831 scope.go:122] "RemoveContainer" containerID="07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"
Oct 31 04:17:38.913200 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:38.913177    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2\": container with ID starting with 07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2 not found: ID does not exist" containerID="07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"
Oct 31 04:17:38.913260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:38.913206    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2"} err="failed to get container status \"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2\": rpc error: code = NotFound desc = could not find container \"07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2\": container with ID starting with 07e1f22ded7f9bfe3fe9d996f0acc1af6461832fd0ca3ee29817e244b9bac3c2 not found: ID does not exist"
Oct 31 04:17:39.041730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:39.041691    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:17:39.087828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:39.087794    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e2a54d-openshift-default-597d99bc45-pkw4n"]
Oct 31 04:17:40.136585 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:40.136550    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" path="/var/lib/kubelet/pods/784e2fcf-7f87-4d52-9569-92918e2caf8d/volumes"
Oct 31 04:17:42.546948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.546908    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:17:42.547559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.547194    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" podUID="b2b081d0-dfb9-4739-9366-fea4bc734067" containerName="maas-api" containerID="cri-o://3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7" gracePeriod=30
Oct 31 04:17:42.856946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.856919    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:17:42.921327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.921288    5831 generic.go:356] "Generic (PLEG): container finished" podID="b2b081d0-dfb9-4739-9366-fea4bc734067" containerID="3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7" exitCode=0
Oct 31 04:17:42.921509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.921339    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"
Oct 31 04:17:42.921509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.921369    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" event={"ID":"b2b081d0-dfb9-4739-9366-fea4bc734067","Type":"ContainerDied","Data":"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"}
Oct 31 04:17:42.921509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.921429    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg" event={"ID":"b2b081d0-dfb9-4739-9366-fea4bc734067","Type":"ContainerDied","Data":"132bf7a9e6a5130834b65297f61553a59d7506bad5dc9e3f62b649692f542b2b"}
Oct 31 04:17:42.921509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:42.921451    5831 scope.go:122] "RemoveContainer" containerID="3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"
Oct 31 04:17:43.008803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.003707    5831 scope.go:122] "RemoveContainer" containerID="3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"
Oct 31 04:17:43.016764 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:43.016604    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7\": container with ID starting with 3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7 not found: ID does not exist" containerID="3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"
Oct 31 04:17:43.017047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.017013    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7"} err="failed to get container status \"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7\": rpc error: code = NotFound desc = could not find container \"3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7\": container with ID starting with 3dc9466d284c3094abc604d96f3f11741336c4a73b8452670c7f8842c8da2af7 not found: ID does not exist"
Oct 31 04:17:43.022013 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.021954    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") pod \"b2b081d0-dfb9-4739-9366-fea4bc734067\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") "
Oct 31 04:17:43.022284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.022261    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\") pod \"b2b081d0-dfb9-4739-9366-fea4bc734067\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") "
Oct 31 04:17:43.022736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.022718    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\") pod \"b2b081d0-dfb9-4739-9366-fea4bc734067\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") "
Oct 31 04:17:43.023038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.023022    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\" (UniqueName: \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\") pod \"b2b081d0-dfb9-4739-9366-fea4bc734067\" (UID: \"b2b081d0-dfb9-4739-9366-fea4bc734067\") "
Oct 31 04:17:43.023710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.023500    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca" pod "b2b081d0-dfb9-4739-9366-fea4bc734067" (UID: "b2b081d0-dfb9-4739-9366-fea4bc734067"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:43.023710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.023681    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b081d0-dfb9-4739-9366-fea4bc734067-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:43.029621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.029473    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls" pod "b2b081d0-dfb9-4739-9366-fea4bc734067" (UID: "b2b081d0-dfb9-4739-9366-fea4bc734067"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:43.048094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.047775    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555" pod "b2b081d0-dfb9-4739-9366-fea4bc734067" (UID: "b2b081d0-dfb9-4739-9366-fea4bc734067"). InnerVolumeSpecName "kube-api-access-2x555". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:43.048794 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.048747    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs" pod "b2b081d0-dfb9-4739-9366-fea4bc734067" (UID: "b2b081d0-dfb9-4739-9366-fea4bc734067"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:43.049675 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:17:43.049641    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:17:43.127660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.125618    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x555\" (UniqueName: \"kubernetes.io/projected/b2b081d0-dfb9-4739-9366-fea4bc734067-kube-api-access-2x555\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:43.127660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.125662    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:43.127660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.125678    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b081d0-dfb9-4739-9366-fea4bc734067-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:43.337812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.337774    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:17:43.383523 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:43.383443    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83f052-59c596497d-2pbdg"]
Oct 31 04:17:44.135853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:44.135773    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b081d0-dfb9-4739-9366-fea4bc734067" path="/var/lib/kubelet/pods/b2b081d0-dfb9-4739-9366-fea4bc734067/volumes"
Oct 31 04:17:57.070074 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:57.070027    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:17:57.205799 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:17:57.205755    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:18:00.753601 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:00.752714    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hxx62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:18:00.753601 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:00.752838    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:18:02.071985 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.071915    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" podUID="24ce1685-c126-4d48-b9a0-d42efec02ec8" containerName="payload-processing" containerID="cri-o://4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240" gracePeriod=25
Oct 31 04:18:02.206951 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.206905    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" podUID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" containerName="payload-pre-processing" containerID="cri-o://32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55" gracePeriod=25
Oct 31 04:18:02.372627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.372599    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:18:02.409206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.409174    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\") pod \"24ce1685-c126-4d48-b9a0-d42efec02ec8\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") "
Oct 31 04:18:02.409432 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.409414    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\" (UniqueName: \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\") pod \"24ce1685-c126-4d48-b9a0-d42efec02ec8\" (UID: \"24ce1685-c126-4d48-b9a0-d42efec02ec8\") "
Oct 31 04:18:02.410488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.410436    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume" pod "24ce1685-c126-4d48-b9a0-d42efec02ec8" (UID: "24ce1685-c126-4d48-b9a0-d42efec02ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:02.412810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.412782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m" pod "24ce1685-c126-4d48-b9a0-d42efec02ec8" (UID: "24ce1685-c126-4d48-b9a0-d42efec02ec8"). InnerVolumeSpecName "kube-api-access-flq6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:02.494185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.494159    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:18:02.511111 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511051    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\") pod \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") "
Oct 31 04:18:02.511329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511132    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\" (UniqueName: \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\") pod \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\" (UID: \"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8\") "
Oct 31 04:18:02.511861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511567    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume" pod "c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" (UID: "c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:02.511861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511834    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flq6m\" (UniqueName: \"kubernetes.io/projected/24ce1685-c126-4d48-b9a0-d42efec02ec8-kube-api-access-flq6m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:02.511861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511859    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:02.512159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.511874    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce1685-c126-4d48-b9a0-d42efec02ec8-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:02.514974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.514945    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx" pod "c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" (UID: "c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8"). InnerVolumeSpecName "kube-api-access-c4tzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:02.613504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:02.613414    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4tzx\" (UniqueName: \"kubernetes.io/projected/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8-kube-api-access-c4tzx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:03.187857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.187822    5831 generic.go:356] "Generic (PLEG): container finished" podID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" containerID="32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55" exitCode=0
Oct 31 04:18:03.188325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.187871    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"
Oct 31 04:18:03.188325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.187926    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" event={"ID":"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8","Type":"ContainerDied","Data":"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"}
Oct 31 04:18:03.188325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.187993    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf" event={"ID":"c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8","Type":"ContainerDied","Data":"8c97095a956f427a3354b595d95bdd73492e157a0a8c44f4a8d6bc1e981908d0"}
Oct 31 04:18:03.188325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.188025    5831 scope.go:122] "RemoveContainer" containerID="32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"
Oct 31 04:18:03.189983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.189961    5831 generic.go:356] "Generic (PLEG): container finished" podID="24ce1685-c126-4d48-b9a0-d42efec02ec8" containerID="4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240" exitCode=0
Oct 31 04:18:03.190121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.190016    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"
Oct 31 04:18:03.190121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.190022    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" event={"ID":"24ce1685-c126-4d48-b9a0-d42efec02ec8","Type":"ContainerDied","Data":"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"}
Oct 31 04:18:03.190121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.190062    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n" event={"ID":"24ce1685-c126-4d48-b9a0-d42efec02ec8","Type":"ContainerDied","Data":"3a62ddf93b467495af40710262732f3e5fe9e303510bd6f866ec8050fe2e9061"}
Oct 31 04:18:03.202214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.202182    5831 scope.go:122] "RemoveContainer" containerID="32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"
Oct 31 04:18:03.202637 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:03.202611    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55\": container with ID starting with 32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55 not found: ID does not exist" containerID="32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"
Oct 31 04:18:03.202693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.202653    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55"} err="failed to get container status \"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55\": rpc error: code = NotFound desc = could not find container \"32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55\": container with ID starting with 32f01fd36f024f5800e45fff7a551de017589990ee647807b71821cb101c7d55 not found: ID does not exist"
Oct 31 04:18:03.202693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.202683    5831 scope.go:122] "RemoveContainer" containerID="4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"
Oct 31 04:18:03.215421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.215387    5831 scope.go:122] "RemoveContainer" containerID="4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"
Oct 31 04:18:03.215814 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:03.215792    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240\": container with ID starting with 4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240 not found: ID does not exist" containerID="4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"
Oct 31 04:18:03.215913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.215821    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240"} err="failed to get container status \"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240\": rpc error: code = NotFound desc = could not find container \"4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240\": container with ID starting with 4d183f106aa7883640feeee66163a92a0cc89a228657203a9bbaa9a50c1c9240 not found: ID does not exist"
Oct 31 04:18:03.343921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.343842    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:18:03.393981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.393071    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f30f2-66f76ccb97-qnt8n"]
Oct 31 04:18:03.458558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.458500    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:18:03.461700 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:03.461671    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f30f2-fb67659dc-57djf"]
Oct 31 04:18:04.138701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:04.138664    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ce1685-c126-4d48-b9a0-d42efec02ec8" path="/var/lib/kubelet/pods/24ce1685-c126-4d48-b9a0-d42efec02ec8/volumes"
Oct 31 04:18:04.139324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:04.139300    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" path="/var/lib/kubelet/pods/c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8/volumes"
Oct 31 04:18:07.976548 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:07.976493    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:18:07.977030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:07.976758    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy" containerID="cri-o://c3ed032fa5fe04d4eaa9aa6c89a5c43651fa42ba3d6f949b63f641af3cf1ceec" gracePeriod=30
Oct 31 04:18:12.979223 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:12.979180    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:18:13.270806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.270768    5831 generic.go:356] "Generic (PLEG): container finished" podID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerID="c3ed032fa5fe04d4eaa9aa6c89a5c43651fa42ba3d6f949b63f641af3cf1ceec" exitCode=0
Oct 31 04:18:13.270984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.270815    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerDied","Data":"c3ed032fa5fe04d4eaa9aa6c89a5c43651fa42ba3d6f949b63f641af3cf1ceec"}
Oct 31 04:18:13.270984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.270859    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql" event={"ID":"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7","Type":"ContainerDied","Data":"9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e"}
Oct 31 04:18:13.270984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.270884    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f31e5b7b0d5065bc154398f81ad3b3cc690ad5a0bd16bb5e1e82775ad03258e"
Oct 31 04:18:13.270984 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.270902    5831 scope.go:122] "RemoveContainer" containerID="bd6a7446a13645ded420789fbe9b9577b891a34bae0c07c20c8897cf40d4ce06"
Oct 31 04:18:13.274607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.274587    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:18:13.356853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.356821    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.356868    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.356941    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.356987    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357033    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357071    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357133    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357168    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:13.357507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357183    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:13.357507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357180    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357266    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\") pod \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\" (UID: \"c2c34cb3-a037-49e5-90e1-b02eb0b11bc7\") "
Oct 31 04:18:13.357507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357321    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:13.357879 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357594    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:13.357879 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.357647    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:13.358165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.358095    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.358165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.358116    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.358165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.358134    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.358165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.358148    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.358403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.358167    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.359723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.359693    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:18:13.359832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.359782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:13.359981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.359966    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "kube-api-access-jn6cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:13.360028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.359972    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy" pod "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" (UID: "c2c34cb3-a037-49e5-90e1-b02eb0b11bc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:13.460001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.459962    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.460175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.460006    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn6cq\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-kube-api-access-jn6cq\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.460175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.460023    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:13.460175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:13.460036    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:14.113912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:14.113865    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:18:14.279642 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:14.279612    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"
Oct 31 04:18:14.491615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:14.491509    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:18:14.495584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:14.495555    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e53098-openshift-default-75b4bbb4c-7mlql"]
Oct 31 04:18:16.140929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:16.140897    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" path="/var/lib/kubelet/pods/c2c34cb3-a037-49e5-90e1-b02eb0b11bc7/volumes"
Oct 31 04:18:17.051887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.051851    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:18:17.052480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052466    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052481    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052491    5831 cpu_manager.go:414] "containerMap: removing container" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052496    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052514    5831 cpu_manager.go:414] "containerMap: removing container" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052519    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052539    5831 cpu_manager.go:414] "containerMap: removing container" podUID="24ce1685-c126-4d48-b9a0-d42efec02ec8" containerName="payload-processing"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052544    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ce1685-c126-4d48-b9a0-d42efec02ec8" containerName="payload-processing"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052551    5831 cpu_manager.go:414] "containerMap: removing container" podUID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" containerName="payload-processing"
Oct 31 04:18:17.052560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052557    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" containerName="payload-processing"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052576    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052581    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052591    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052596    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052603    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" containerName="payload-pre-processing"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052607    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" containerName="payload-pre-processing"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052617    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052622    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052633    5831 cpu_manager.go:414] "containerMap: removing container" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052637    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052655    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fa32a48f-34c7-4ff9-8277-fc4181a231cc" containerName="revoke-keys"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052659    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa32a48f-34c7-4ff9-8277-fc4181a231cc" containerName="revoke-keys"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052676    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b2b081d0-dfb9-4739-9366-fea4bc734067" containerName="maas-api"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052681    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b081d0-dfb9-4739-9366-fea4bc734067" containerName="maas-api"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052769    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052779    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b9f74f-fb2a-4c2d-a1a4-beac82ecc6b8" containerName="payload-pre-processing"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052787    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052797    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052819    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa32a48f-34c7-4ff9-8277-fc4181a231cc" containerName="revoke-keys"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052834    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b081d0-dfb9-4739-9366-fea4bc734067" containerName="maas-api"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052844    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e9fbeb-0d6d-4a8b-9a52-7a822613e8dc" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052857    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="784e2fcf-7f87-4d52-9569-92918e2caf8d" containerName="istio-proxy"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052864    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ce1685-c126-4d48-b9a0-d42efec02ec8" containerName="payload-processing"
Oct 31 04:18:17.052918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.052870    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d1c8c9-f286-4cd5-8863-fa926efb7d65" containerName="payload-processing"
Oct 31 04:18:17.058298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.058277    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224473    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224510    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224662    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224687    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224714    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqspm\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224892    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.224966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224918    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.225152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.224994    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.225152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.225065    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.254018 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.253983    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:18:17.326701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326615    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.326701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326681    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.326938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326719    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.326938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326737    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.326938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326824    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.326938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326855    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327190 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.326893    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqspm\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327190 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327081    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327223    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327292    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327321    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327460    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327482    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.327773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.327752    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.329465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.329438    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.330025 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.330008    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.380321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.380284    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqspm\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.380484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.380404    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qcfrt\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:17.672972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:17.672882    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:18.097261 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.097229    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:18:18.099280 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:18:18.099255    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe072865_11e4_4d5f_9e2a_e21e416aea37.slice/crio-07180e130675e890fb19d6a5971d32f9bff20028afb1b059c69d91241a01bf50 WatchSource:0}: Error finding container 07180e130675e890fb19d6a5971d32f9bff20028afb1b059c69d91241a01bf50: Status 404 returned error can't find the container with id 07180e130675e890fb19d6a5971d32f9bff20028afb1b059c69d91241a01bf50
Oct 31 04:18:18.101969 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.101936    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:18.102073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.102000    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:18.102073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.102030    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:18.317928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.317849    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" event={"ID":"be072865-11e4-4d5f-9e2a-e21e416aea37","Type":"ContainerStarted","Data":"0ee47971753e31ecbed19e7e6f7a939278c33723e0388cce254bbdaadd2678aa"}
Oct 31 04:18:18.317928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.317916    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" event={"ID":"be072865-11e4-4d5f-9e2a-e21e416aea37","Type":"ContainerStarted","Data":"07180e130675e890fb19d6a5971d32f9bff20028afb1b059c69d91241a01bf50"}
Oct 31 04:18:18.427172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.427061    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podStartSLOduration=1.427042843 podStartE2EDuration="1.427042843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:18.426022774 +0000 UTC m=+4248.440584493" watchObservedRunningTime="2026-10-31 04:18:18.427042843 +0000 UTC m=+4248.441604557"
Oct 31 04:18:18.673614 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:18.673575    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:19.114249 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.114194    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" podUID="f9b0775f-f340-4aae-b789-58c1ed170160" containerName="payload-processing" containerID="cri-o://9a41ed38fd4de855264a73b5d3e15425af09771d7f1c3cf6b7ba4278602dbd5c" gracePeriod=25
Oct 31 04:18:19.330822 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.330794    5831 generic.go:356] "Generic (PLEG): container finished" podID="f9b0775f-f340-4aae-b789-58c1ed170160" containerID="9a41ed38fd4de855264a73b5d3e15425af09771d7f1c3cf6b7ba4278602dbd5c" exitCode=0
Oct 31 04:18:19.331226 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.330867    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" event={"ID":"f9b0775f-f340-4aae-b789-58c1ed170160","Type":"ContainerDied","Data":"9a41ed38fd4de855264a73b5d3e15425af09771d7f1c3cf6b7ba4278602dbd5c"}
Oct 31 04:18:19.436710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.436684    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:18:19.560794 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.560757    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\") pod \"f9b0775f-f340-4aae-b789-58c1ed170160\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") "
Oct 31 04:18:19.560990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.560901    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\" (UniqueName: \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\") pod \"f9b0775f-f340-4aae-b789-58c1ed170160\" (UID: \"f9b0775f-f340-4aae-b789-58c1ed170160\") "
Oct 31 04:18:19.561223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.561200    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume" pod "f9b0775f-f340-4aae-b789-58c1ed170160" (UID: "f9b0775f-f340-4aae-b789-58c1ed170160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:19.561381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.561366    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0775f-f340-4aae-b789-58c1ed170160-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:19.563247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.563223    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw" pod "f9b0775f-f340-4aae-b789-58c1ed170160" (UID: "f9b0775f-f340-4aae-b789-58c1ed170160"). InnerVolumeSpecName "kube-api-access-gv4zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:19.662876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.662794    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4zw\" (UniqueName: \"kubernetes.io/projected/f9b0775f-f340-4aae-b789-58c1ed170160-kube-api-access-gv4zw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:19.675438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.675407    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:19.675624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:19.675457    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:20.340385 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.340357    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"
Oct 31 04:18:20.340950 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.340387    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9" event={"ID":"f9b0775f-f340-4aae-b789-58c1ed170160","Type":"ContainerDied","Data":"cef9814efdffb8447ad1ffcac1be1ef75a39a3f5a589698f641fa36efebc0e61"}
Oct 31 04:18:20.340950 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.340451    5831 scope.go:122] "RemoveContainer" containerID="9a41ed38fd4de855264a73b5d3e15425af09771d7f1c3cf6b7ba4278602dbd5c"
Oct 31 04:18:20.507589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.507554    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:18:20.510754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.510730    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e679fe-5cf96d4ddc-r49c9"]
Oct 31 04:18:20.673217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.673180    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:20.673405 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:20.673257    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:21.674087 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:21.674048    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:21.674472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:21.674110    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:22.136105 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:22.136066    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b0775f-f340-4aae-b789-58c1ed170160" path="/var/lib/kubelet/pods/f9b0775f-f340-4aae-b789-58c1ed170160/volumes"
Oct 31 04:18:22.673212 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:22.673177    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:22.673408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:22.673236    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:23.673255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:23.673217    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.673763 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:23.673281    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:24.673611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:24.673571    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.674064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:24.673643    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:25.673851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:25.673806    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qcfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.674313 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:25.673895    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Oct 31 04:18:26.678980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:26.678944    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:26.679403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:26.679360    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:26.680269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:26.680243    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:18:32.161944 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.161910    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:18:32.162521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.162506    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f9b0775f-f340-4aae-b789-58c1ed170160" containerName="payload-processing"
Oct 31 04:18:32.162521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.162521    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b0775f-f340-4aae-b789-58c1ed170160" containerName="payload-processing"
Oct 31 04:18:32.162780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.162667    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9745416-29f8-4484-9cad-b205d8ecbae1" containerName="istio-proxy"
Oct 31 04:18:32.162780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.162675    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c34cb3-a037-49e5-90e1-b02eb0b11bc7" containerName="istio-proxy"
Oct 31 04:18:32.162780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.162683    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b0775f-f340-4aae-b789-58c1ed170160" containerName="payload-processing"
Oct 31 04:18:32.166443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.166422    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261209 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261170    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261493    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261569    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261605    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261652    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261714    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261756    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261789    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.261914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.261849    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhv2\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.266758 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.266728    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:18:32.365632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.365589    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.365944 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.365916    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.366122 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.366101    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.366292 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.366273    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhv2\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.366561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.366518    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.366853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.366820    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.367021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.367001    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.367198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.367171    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.367373 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.367344    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.374852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.370795    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.379270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.373991    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.379499 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.374571    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.380462 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.375076    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.380657 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.375313    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.380786 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.375928    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.386483 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.386055    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.428421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.428392    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.429256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.429234    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhv2\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\") pod \"maas-default-gateway-openshift-default-6886b96f9-jlkc2\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.481447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.481411    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:32.906910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.906877    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:18:32.909100 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:18:32.909068    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1123a65b_1994_4d76_82c5_0fe4836596f0.slice/crio-273d230411da407f8e3cf05be02e9dc892faff9eab38387e15be30e81559c5a0 WatchSource:0}: Error finding container 273d230411da407f8e3cf05be02e9dc892faff9eab38387e15be30e81559c5a0: Status 404 returned error can't find the container with id 273d230411da407f8e3cf05be02e9dc892faff9eab38387e15be30e81559c5a0
Oct 31 04:18:32.912375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.912338    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:32.912475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.912410    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:32.912475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:32.912438    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:18:33.457420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.457379    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" event={"ID":"1123a65b-1994-4d76-82c5-0fe4836596f0","Type":"ContainerStarted","Data":"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"}
Oct 31 04:18:33.457921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.457444    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" event={"ID":"1123a65b-1994-4d76-82c5-0fe4836596f0","Type":"ContainerStarted","Data":"273d230411da407f8e3cf05be02e9dc892faff9eab38387e15be30e81559c5a0"}
Oct 31 04:18:33.481588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.481561    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:33.483372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.483343    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:33.483543 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.483412    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:33.567261 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:33.567205    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podStartSLOduration=1.567188231 podStartE2EDuration="1.567188231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:33.565881254 +0000 UTC m=+4263.580442975" watchObservedRunningTime="2026-10-31 04:18:33.567188231 +0000 UTC m=+4263.581749942"
Oct 31 04:18:34.493554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:34.489937    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:34.493554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:34.490008    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:35.501553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:35.500017    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:35.501553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:35.500106    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:36.484558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:36.482983    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:36.484558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:36.483044    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:36.868973 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:36.868886    5831 scope.go:122] "RemoveContainer" containerID="c3ed032fa5fe04d4eaa9aa6c89a5c43651fa42ba3d6f949b63f641af3cf1ceec"
Oct 31 04:18:37.484556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:37.483905    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:37.484556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:37.483964    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:38.483570 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:38.482952    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:38.483570 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:38.483018    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:39.482765 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:39.482723    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:39.482993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:39.482793    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:39.670409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:39.670375    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:18:40.482738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:40.482704    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jlkc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Oct 31 04:18:40.482936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:40.482760    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Oct 31 04:18:41.617547 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:41.616827    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:41.618134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:41.618109    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:41.619136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:41.619113    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:18:42.979416 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:42.979375    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:18:49.066261 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:49.065946    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:18:49.067959 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:49.067037    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" containerID="cri-o://1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717" gracePeriod=30
Oct 31 04:18:50.217288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:50.217252    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:18:50.382732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:50.382694    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:18:53.601425 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:53.601386    5831 patch_prober.go:29] interesting pod/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:18:53.601938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:53.601448    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:18:54.387015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.386986    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:18:54.506381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506350    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506414    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506567    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506653    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506691    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506724    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506738    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:54.506981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506786    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.506981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.506851    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.507095 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507011    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\") pod \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\" (UID: \"7cb0604b-7877-4362-b83d-a3eb3e62cb9a\") "
Oct 31 04:18:54.507353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507008    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:54.507456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507035    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:54.507456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507262    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:54.507601 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507452    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:54.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507651    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507677    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507694    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507709    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.508037 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.507758    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.509434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.509404    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "kube-api-access-5swj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:54.509631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.509608    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:18:54.509775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.509756    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:54.509840 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.509817    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token" pod "7cb0604b-7877-4362-b83d-a3eb3e62cb9a" (UID: "7cb0604b-7877-4362-b83d-a3eb3e62cb9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:54.609077 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.608972    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.609077 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.609015    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.609077 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.609032    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.609077 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.609047    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swj9\" (UniqueName: \"kubernetes.io/projected/7cb0604b-7877-4362-b83d-a3eb3e62cb9a-kube-api-access-5swj9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:54.669830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.669792    5831 generic.go:356] "Generic (PLEG): container finished" podID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerID="1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717" exitCode=0
Oct 31 04:18:54.670033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.669850    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"
Oct 31 04:18:54.670033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.669885    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerDied","Data":"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"}
Oct 31 04:18:54.670033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.669942    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4" event={"ID":"7cb0604b-7877-4362-b83d-a3eb3e62cb9a","Type":"ContainerDied","Data":"125d9bb0e1704f23cdeaa34c2aab9ba94406782ce3ed8ba1bda461a1e7aa61e8"}
Oct 31 04:18:54.670033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.669968    5831 scope.go:122] "RemoveContainer" containerID="1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"
Oct 31 04:18:54.683918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.683888    5831 scope.go:122] "RemoveContainer" containerID="3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"
Oct 31 04:18:54.697883 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.697862    5831 scope.go:122] "RemoveContainer" containerID="1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"
Oct 31 04:18:54.698245 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:54.698224    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717\": container with ID starting with 1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717 not found: ID does not exist" containerID="1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"
Oct 31 04:18:54.698354 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.698253    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717"} err="failed to get container status \"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717\": rpc error: code = NotFound desc = could not find container \"1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717\": container with ID starting with 1a42c8a2995715f47ea76352b144042c27e8a69d89eb6c7b8729ba500e4eb717 not found: ID does not exist"
Oct 31 04:18:54.698354 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.698274    5831 scope.go:122] "RemoveContainer" containerID="3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"
Oct 31 04:18:54.698622 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:54.698591    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a\": container with ID starting with 3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a not found: ID does not exist" containerID="3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"
Oct 31 04:18:54.698707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.698631    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a"} err="failed to get container status \"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a\": rpc error: code = NotFound desc = could not find container \"3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a\": container with ID starting with 3ee35562fe6dcff60bf097bea15f6c2b0a7d6b8375e46dd094c01162dde8a63a not found: ID does not exist"
Oct 31 04:18:54.826587 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.826556    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:18:54.831283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:54.831252    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-243a29-openshift-default-596fff8f5-zn2w4"]
Oct 31 04:18:55.170712 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.170680    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"]
Oct 31 04:18:55.171336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.171321    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:18:55.171402 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.171338    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:18:55.171491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.171481    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:18:55.171547 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.171499    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:18:55.176902 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.176883    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.216363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.216328    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vr6\" (UniqueName: \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.216575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.216488    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.216713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.216692    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.217889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.217858    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" podUID="c4aade3b-cbb1-4259-836c-e03188900fac" containerName="payload-processing" containerID="cri-o://eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56" gracePeriod=25
Oct 31 04:18:55.226315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.226289    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:18:55.226939 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.226922    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:18:55.228027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.228002    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"]
Oct 31 04:18:55.317690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.317651    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vr6\" (UniqueName: \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.317860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.317756    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.317860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.317815    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.318502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.318478    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.320353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.320323    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.371377 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.371344    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vr6\" (UniqueName: \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\") pod \"maas-api-revoke-sub-5adf9b8d-ldn64\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.383158 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.383114    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" podUID="b73ed55d-4659-4fd9-af10-7c7dbf189467" containerName="payload-pre-processing" containerID="cri-o://0f0ef20d051704b6853cc7d82e27b554ba7589fb84ca9692e0e1bdec5cb2dfad" gracePeriod=25
Oct 31 04:18:55.493047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.493021    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:55.506008 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.505894    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:18:55.520030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.519996    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\") pod \"c4aade3b-cbb1-4259-836c-e03188900fac\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") "
Oct 31 04:18:55.520170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.520062    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\" (UniqueName: \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\") pod \"c4aade3b-cbb1-4259-836c-e03188900fac\" (UID: \"c4aade3b-cbb1-4259-836c-e03188900fac\") "
Oct 31 04:18:55.520897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.520869    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume" pod "c4aade3b-cbb1-4259-836c-e03188900fac" (UID: "c4aade3b-cbb1-4259-836c-e03188900fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:55.525820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.525776    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx" pod "c4aade3b-cbb1-4259-836c-e03188900fac" (UID: "c4aade3b-cbb1-4259-836c-e03188900fac"). InnerVolumeSpecName "kube-api-access-l7kbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:55.625256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.624916    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kbx\" (UniqueName: \"kubernetes.io/projected/c4aade3b-cbb1-4259-836c-e03188900fac-kube-api-access-l7kbx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:55.625256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.624950    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4aade3b-cbb1-4259-836c-e03188900fac-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:55.687079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.687041    5831 generic.go:356] "Generic (PLEG): container finished" podID="c4aade3b-cbb1-4259-836c-e03188900fac" containerID="eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56" exitCode=0
Oct 31 04:18:55.687308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.687290    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"
Oct 31 04:18:55.688099 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.688063    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" event={"ID":"c4aade3b-cbb1-4259-836c-e03188900fac","Type":"ContainerDied","Data":"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"}
Oct 31 04:18:55.688242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.688166    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9" event={"ID":"c4aade3b-cbb1-4259-836c-e03188900fac","Type":"ContainerDied","Data":"04304b79d9dc9742649361f311bff6baba6aa7fa5403e762d5cbbbe84cd0b0fd"}
Oct 31 04:18:55.688242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.688200    5831 scope.go:122] "RemoveContainer" containerID="eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"
Oct 31 04:18:55.691702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.691678    5831 generic.go:356] "Generic (PLEG): container finished" podID="b73ed55d-4659-4fd9-af10-7c7dbf189467" containerID="0f0ef20d051704b6853cc7d82e27b554ba7589fb84ca9692e0e1bdec5cb2dfad" exitCode=0
Oct 31 04:18:55.691865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.691847    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" event={"ID":"b73ed55d-4659-4fd9-af10-7c7dbf189467","Type":"ContainerDied","Data":"0f0ef20d051704b6853cc7d82e27b554ba7589fb84ca9692e0e1bdec5cb2dfad"}
Oct 31 04:18:55.696458 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.696410    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:18:55.712335 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.712302    5831 scope.go:122] "RemoveContainer" containerID="eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"
Oct 31 04:18:55.712897 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:18:55.712874    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56\": container with ID starting with eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56 not found: ID does not exist" containerID="eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"
Oct 31 04:18:55.713002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.712911    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56"} err="failed to get container status \"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56\": rpc error: code = NotFound desc = could not find container \"eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56\": container with ID starting with eabeb6bfe7f7304079f44f2563038480dabe330dd4f62b2b61be643dd873ab56 not found: ID does not exist"
Oct 31 04:18:55.726454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.726426    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\") pod \"b73ed55d-4659-4fd9-af10-7c7dbf189467\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") "
Oct 31 04:18:55.726885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.726858    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume" pod "b73ed55d-4659-4fd9-af10-7c7dbf189467" (UID: "b73ed55d-4659-4fd9-af10-7c7dbf189467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:55.726988 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.726945    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ed55d-4659-4fd9-af10-7c7dbf189467-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:55.827550 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.827451    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\" (UniqueName: \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\") pod \"b73ed55d-4659-4fd9-af10-7c7dbf189467\" (UID: \"b73ed55d-4659-4fd9-af10-7c7dbf189467\") "
Oct 31 04:18:55.830005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.829977    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb" pod "b73ed55d-4659-4fd9-af10-7c7dbf189467" (UID: "b73ed55d-4659-4fd9-af10-7c7dbf189467"). InnerVolumeSpecName "kube-api-access-dr7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:55.875867 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.875819    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"]
Oct 31 04:18:55.877431 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:18:55.877404    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf157398e_7734_4d01_a21b_9a7a55e2cc85.slice/crio-a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0 WatchSource:0}: Error finding container a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0: Status 404 returned error can't find the container with id a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0
Oct 31 04:18:55.928372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.928341    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7mb\" (UniqueName: \"kubernetes.io/projected/b73ed55d-4659-4fd9-af10-7c7dbf189467-kube-api-access-dr7mb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:55.944539 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.944503    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:18:55.949659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:55.949637    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c42c43-7d4c7fd4d7-f48b9"]
Oct 31 04:18:56.136406 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.136318    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" path="/var/lib/kubelet/pods/7cb0604b-7877-4362-b83d-a3eb3e62cb9a/volumes"
Oct 31 04:18:56.136837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.136814    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4aade3b-cbb1-4259-836c-e03188900fac" path="/var/lib/kubelet/pods/c4aade3b-cbb1-4259-836c-e03188900fac/volumes"
Oct 31 04:18:56.706327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.706284    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64" event={"ID":"f157398e-7734-4d01-a21b-9a7a55e2cc85","Type":"ContainerStarted","Data":"e917113f41d0d3fc5571d21d915f1905a062ac1ef77f1e3d1960b0748237bd70"}
Oct 31 04:18:56.706843 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.706352    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64" event={"ID":"f157398e-7734-4d01-a21b-9a7a55e2cc85","Type":"ContainerStarted","Data":"a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0"}
Oct 31 04:18:56.707965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.707941    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw" event={"ID":"b73ed55d-4659-4fd9-af10-7c7dbf189467","Type":"ContainerDied","Data":"08e1ac2973fe9f38396fa51a2ad77db2db5846d1c53cccb288ac9533aadedea9"}
Oct 31 04:18:56.707965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.707961    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"
Oct 31 04:18:56.708168 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.707992    5831 scope.go:122] "RemoveContainer" containerID="0f0ef20d051704b6853cc7d82e27b554ba7589fb84ca9692e0e1bdec5cb2dfad"
Oct 31 04:18:56.813058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.813003    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64" podStartSLOduration=1.812988635 podStartE2EDuration="1.812988635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:56.811187706 +0000 UTC m=+4286.825749411" watchObservedRunningTime="2026-10-31 04:18:56.812988635 +0000 UTC m=+4286.827550343"
Oct 31 04:18:56.917304 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.917267    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:18:56.920624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:56.920593    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c42c43-65465649bc-c5xxw"]
Oct 31 04:18:57.732121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:57.732090    5831 generic.go:356] "Generic (PLEG): container finished" podID="f157398e-7734-4d01-a21b-9a7a55e2cc85" containerID="e917113f41d0d3fc5571d21d915f1905a062ac1ef77f1e3d1960b0748237bd70" exitCode=0
Oct 31 04:18:57.732620 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:57.732175    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64" event={"ID":"f157398e-7734-4d01-a21b-9a7a55e2cc85","Type":"ContainerDied","Data":"e917113f41d0d3fc5571d21d915f1905a062ac1ef77f1e3d1960b0748237bd70"}
Oct 31 04:18:58.136365 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.136282    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73ed55d-4659-4fd9-af10-7c7dbf189467" path="/var/lib/kubelet/pods/b73ed55d-4659-4fd9-af10-7c7dbf189467/volumes"
Oct 31 04:18:58.902220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.902186    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:18:58.963872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.963841    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\" (UniqueName: \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\") pod \"f157398e-7734-4d01-a21b-9a7a55e2cc85\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") "
Oct 31 04:18:58.964069 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.963917    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\") pod \"f157398e-7734-4d01-a21b-9a7a55e2cc85\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") "
Oct 31 04:18:58.964069 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.964029    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\") pod \"f157398e-7734-4d01-a21b-9a7a55e2cc85\" (UID: \"f157398e-7734-4d01-a21b-9a7a55e2cc85\") "
Oct 31 04:18:58.964347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.964319    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca" pod "f157398e-7734-4d01-a21b-9a7a55e2cc85" (UID: "f157398e-7734-4d01-a21b-9a7a55e2cc85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:18:58.964518 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.964490    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:58.966298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.966272    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca" pod "f157398e-7734-4d01-a21b-9a7a55e2cc85" (UID: "f157398e-7734-4d01-a21b-9a7a55e2cc85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:18:58.966389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:58.966345    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6" pod "f157398e-7734-4d01-a21b-9a7a55e2cc85" (UID: "f157398e-7734-4d01-a21b-9a7a55e2cc85"). InnerVolumeSpecName "kube-api-access-26vr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:59.065275 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:59.065184    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vr6\" (UniqueName: \"kubernetes.io/projected/f157398e-7734-4d01-a21b-9a7a55e2cc85-kube-api-access-26vr6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:59.065275 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:59.065217    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f157398e-7734-4d01-a21b-9a7a55e2cc85-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:18:59.748290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:59.748254    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64" event={"ID":"f157398e-7734-4d01-a21b-9a7a55e2cc85","Type":"ContainerDied","Data":"a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0"}
Oct 31 04:18:59.748477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:59.748310    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7e75e0a043ec7300f8b4294d1690057505815ad69f254400b99e9b92c5cb4f0"
Oct 31 04:18:59.748477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:18:59.748263    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"
Oct 31 04:19:12.978853 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:19:12.978813    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:19:13.405806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.405705    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:19:13.405998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.405949    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" podUID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" containerName="maas-api" containerID="cri-o://b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07" gracePeriod=30
Oct 31 04:19:13.741007 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.740978    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:19:13.873554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.873270    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\" (UniqueName: \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\") pod \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") "
Oct 31 04:19:13.873554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.873377    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\") pod \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") "
Oct 31 04:19:13.873554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.873464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\") pod \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") "
Oct 31 04:19:13.873904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.873635    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") pod \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\" (UID: \"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb\") "
Oct 31 04:19:13.877549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.876914    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls" pod "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" (UID: "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:13.877549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.877405    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca" pod "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" (UID: "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:13.885683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.884464    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs" pod "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" (UID: "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886275    5831 generic.go:356] "Generic (PLEG): container finished" podID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" containerID="b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07" exitCode=0
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886502    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886545    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" event={"ID":"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb","Type":"ContainerDied","Data":"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"}
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886566    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2" pod "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" (UID: "8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb"). InnerVolumeSpecName "kube-api-access-wbkq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886604    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56" event={"ID":"8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb","Type":"ContainerDied","Data":"18d0bb273c4e041ad875f6c42712a02d7917729fd1429aa4af200dafbc2bf0f8"}
Oct 31 04:19:13.886746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.886629    5831 scope.go:122] "RemoveContainer" containerID="b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"
Oct 31 04:19:13.917863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.917826    5831 scope.go:122] "RemoveContainer" containerID="b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"
Oct 31 04:19:13.918368 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:19:13.918300    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07\": container with ID starting with b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07 not found: ID does not exist" containerID="b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"
Oct 31 04:19:13.918368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:13.918338    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07"} err="failed to get container status \"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07\": rpc error: code = NotFound desc = could not find container \"b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07\": container with ID starting with b54d5bc76dbd8e4533677386ab0138ff36a9294b81740465d8b859015b865e07 not found: ID does not exist"
Oct 31 04:19:14.005264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.001405    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:14.005264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.001441    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:14.005264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.001459    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:14.005264 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.001473    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkq2\" (UniqueName: \"kubernetes.io/projected/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb-kube-api-access-wbkq2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:14.040840 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.040802    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:19:14.045140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.045108    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f30f2-b4b49d9b6-86c56"]
Oct 31 04:19:14.136514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:14.136481    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" path="/var/lib/kubelet/pods/8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb/volumes"
Oct 31 04:19:24.758416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.758378    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"]
Oct 31 04:19:24.759160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759142    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" containerName="maas-api"
Oct 31 04:19:24.759217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759163    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" containerName="maas-api"
Oct 31 04:19:24.759217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759189    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b73ed55d-4659-4fd9-af10-7c7dbf189467" containerName="payload-pre-processing"
Oct 31 04:19:24.759217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759194    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73ed55d-4659-4fd9-af10-7c7dbf189467" containerName="payload-pre-processing"
Oct 31 04:19:24.759217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759207    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:19:24.759217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759212    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb0604b-7877-4362-b83d-a3eb3e62cb9a" containerName="istio-proxy"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759225    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c4aade3b-cbb1-4259-836c-e03188900fac" containerName="payload-processing"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759230    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4aade3b-cbb1-4259-836c-e03188900fac" containerName="payload-processing"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759250    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f157398e-7734-4d01-a21b-9a7a55e2cc85" containerName="revoke-keys"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759254    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157398e-7734-4d01-a21b-9a7a55e2cc85" containerName="revoke-keys"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759338    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f157398e-7734-4d01-a21b-9a7a55e2cc85" containerName="revoke-keys"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759349    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8babf2-b3b3-4018-a6bd-c2ef4fe222cb" containerName="maas-api"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759357    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73ed55d-4659-4fd9-af10-7c7dbf189467" containerName="payload-pre-processing"
Oct 31 04:19:24.759384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.759365    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4aade3b-cbb1-4259-836c-e03188900fac" containerName="payload-processing"
Oct 31 04:19:24.764570 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.764519    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.817329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.817295    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"]
Oct 31 04:19:24.857117 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.857082    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:19:24.857300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.857151    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:19:24.859780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.859759    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4zg\" (UniqueName: \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.859936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.859916    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.859993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.859967    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.961075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.961036    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.961253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.961096    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.961253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.961165    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4zg\" (UniqueName: \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.961832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.961801    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:24.963691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:24.963670    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:25.015447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:25.015364    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4zg\" (UniqueName: \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\") pod \"maas-api-revoke-sub-4deb5301-fvh8k\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:25.078767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:25.078725    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:25.496557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:25.490322    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"]
Oct 31 04:19:25.521082 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:19:25.504252    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb39d151_fa52_4341_9cff_d525b4301886.slice/crio-ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566 WatchSource:0}: Error finding container ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566: Status 404 returned error can't find the container with id ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566
Oct 31 04:19:26.050775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:26.050682    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k" event={"ID":"fb39d151-fa52-4341-9cff-d525b4301886","Type":"ContainerStarted","Data":"aede0a8705701aade6f3c1f074a80245f057365e6c9a78ce07d15e7439b2a350"}
Oct 31 04:19:26.060715 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:26.060672    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k" event={"ID":"fb39d151-fa52-4341-9cff-d525b4301886","Type":"ContainerStarted","Data":"ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566"}
Oct 31 04:19:26.170408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:26.170351    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k" podStartSLOduration=2.170333185 podStartE2EDuration="2.170333185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:26.167434735 +0000 UTC m=+4316.181996443" watchObservedRunningTime="2026-10-31 04:19:26.170333185 +0000 UTC m=+4316.184894895"
Oct 31 04:19:27.059410 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:27.059373    5831 generic.go:356] "Generic (PLEG): container finished" podID="fb39d151-fa52-4341-9cff-d525b4301886" containerID="aede0a8705701aade6f3c1f074a80245f057365e6c9a78ce07d15e7439b2a350" exitCode=0
Oct 31 04:19:27.059863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:27.059409    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k" event={"ID":"fb39d151-fa52-4341-9cff-d525b4301886","Type":"ContainerDied","Data":"aede0a8705701aade6f3c1f074a80245f057365e6c9a78ce07d15e7439b2a350"}
Oct 31 04:19:28.252627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.252602    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:28.421819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.421715    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\") pod \"fb39d151-fa52-4341-9cff-d525b4301886\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") "
Oct 31 04:19:28.422064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.421938    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\" (UniqueName: \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\") pod \"fb39d151-fa52-4341-9cff-d525b4301886\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") "
Oct 31 04:19:28.422064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.421967    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\") pod \"fb39d151-fa52-4341-9cff-d525b4301886\" (UID: \"fb39d151-fa52-4341-9cff-d525b4301886\") "
Oct 31 04:19:28.422468 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.422408    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca" pod "fb39d151-fa52-4341-9cff-d525b4301886" (UID: "fb39d151-fa52-4341-9cff-d525b4301886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:28.424171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.424140    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg" pod "fb39d151-fa52-4341-9cff-d525b4301886" (UID: "fb39d151-fa52-4341-9cff-d525b4301886"). InnerVolumeSpecName "kube-api-access-sc4zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:28.424171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.424155    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca" pod "fb39d151-fa52-4341-9cff-d525b4301886" (UID: "fb39d151-fa52-4341-9cff-d525b4301886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:28.523311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.523277    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4zg\" (UniqueName: \"kubernetes.io/projected/fb39d151-fa52-4341-9cff-d525b4301886-kube-api-access-sc4zg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:28.523311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.523306    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:28.523311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:28.523317    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb39d151-fa52-4341-9cff-d525b4301886-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:29.075795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:29.075766    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"
Oct 31 04:19:29.075795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:29.075778    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k" event={"ID":"fb39d151-fa52-4341-9cff-d525b4301886","Type":"ContainerDied","Data":"ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566"}
Oct 31 04:19:29.076022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:29.075822    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9d383e38acba0166e71ec29df1300d53fd08cc2edea923f4daebb3be531566"
Oct 31 04:19:30.617514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:30.617468    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:19:42.980295 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:19:42.980257    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:19:59.577393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.577351    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"]
Oct 31 04:19:59.580113 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.578345    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fb39d151-fa52-4341-9cff-d525b4301886" containerName="revoke-keys"
Oct 31 04:19:59.580113 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.578365    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb39d151-fa52-4341-9cff-d525b4301886" containerName="revoke-keys"
Oct 31 04:19:59.580113 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.578558    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb39d151-fa52-4341-9cff-d525b4301886" containerName="revoke-keys"
Oct 31 04:19:59.582453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.582433    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.646049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.645049    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.657617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.657577    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.657910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.657889    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzlk\" (UniqueName: \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.661421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.661392    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"]
Oct 31 04:19:59.685556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.682796    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:19:59.685556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.682993    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:19:59.772578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.759311    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.772578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.759370    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzlk\" (UniqueName: \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.772578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.759484    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.778472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.778434    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.799838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.799775    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.839671 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.839437    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzlk\" (UniqueName: \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\") pod \"maas-api-revoke-sub-22274710-25m96\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:19:59.897307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:19:59.897258    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:20:00.313468 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:00.313425    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"]
Oct 31 04:20:00.429735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:00.429694    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96" event={"ID":"9bf7a0bc-5b35-406d-92ec-784510e5dede","Type":"ContainerStarted","Data":"a94fec6adc3902268c1f95299dc198826fbb208a14aedc51086ca7a18ac9606a"}
Oct 31 04:20:01.471613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:01.471367    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96" event={"ID":"9bf7a0bc-5b35-406d-92ec-784510e5dede","Type":"ContainerStarted","Data":"d1414b48785cb55f55f0c4f892f54acd9548972ae5a6c98ab8b462451453ea11"}
Oct 31 04:20:01.617554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:01.613304    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96" podStartSLOduration=2.613282057 podStartE2EDuration="2.613282057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:01.590019523 +0000 UTC m=+4351.604581229" watchObservedRunningTime="2026-10-31 04:20:01.613282057 +0000 UTC m=+4351.627843765"
Oct 31 04:20:02.531598 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:02.531539    5831 generic.go:356] "Generic (PLEG): container finished" podID="9bf7a0bc-5b35-406d-92ec-784510e5dede" containerID="d1414b48785cb55f55f0c4f892f54acd9548972ae5a6c98ab8b462451453ea11" exitCode=0
Oct 31 04:20:02.532447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:02.532366    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96" event={"ID":"9bf7a0bc-5b35-406d-92ec-784510e5dede","Type":"ContainerDied","Data":"d1414b48785cb55f55f0c4f892f54acd9548972ae5a6c98ab8b462451453ea11"}
Oct 31 04:20:03.751219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.751183    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:20:03.857256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.857155    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\") pod \"9bf7a0bc-5b35-406d-92ec-784510e5dede\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") "
Oct 31 04:20:03.857256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.857247    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\" (UniqueName: \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\") pod \"9bf7a0bc-5b35-406d-92ec-784510e5dede\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") "
Oct 31 04:20:03.857513 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.857361    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\") pod \"9bf7a0bc-5b35-406d-92ec-784510e5dede\" (UID: \"9bf7a0bc-5b35-406d-92ec-784510e5dede\") "
Oct 31 04:20:03.864597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.861009    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca" pod "9bf7a0bc-5b35-406d-92ec-784510e5dede" (UID: "9bf7a0bc-5b35-406d-92ec-784510e5dede"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:03.864597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.863863    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk" pod "9bf7a0bc-5b35-406d-92ec-784510e5dede" (UID: "9bf7a0bc-5b35-406d-92ec-784510e5dede"). InnerVolumeSpecName "kube-api-access-8qzlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:03.881894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.881855    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca" pod "9bf7a0bc-5b35-406d-92ec-784510e5dede" (UID: "9bf7a0bc-5b35-406d-92ec-784510e5dede"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:03.958948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.958908    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:03.958948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.958951    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qzlk\" (UniqueName: \"kubernetes.io/projected/9bf7a0bc-5b35-406d-92ec-784510e5dede-kube-api-access-8qzlk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:03.959223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:03.958970    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a0bc-5b35-406d-92ec-784510e5dede-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:04.080811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:04.080775    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 04:20:04.549599 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:04.549522    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96" event={"ID":"9bf7a0bc-5b35-406d-92ec-784510e5dede","Type":"ContainerDied","Data":"a94fec6adc3902268c1f95299dc198826fbb208a14aedc51086ca7a18ac9606a"}
Oct 31 04:20:04.549793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:04.549610    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94fec6adc3902268c1f95299dc198826fbb208a14aedc51086ca7a18ac9606a"
Oct 31 04:20:04.549793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:04.549566    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"
Oct 31 04:20:12.136804 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.136768    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"]
Oct 31 04:20:12.137507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.137488    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9bf7a0bc-5b35-406d-92ec-784510e5dede" containerName="revoke-keys"
Oct 31 04:20:12.137581 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.137509    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf7a0bc-5b35-406d-92ec-784510e5dede" containerName="revoke-keys"
Oct 31 04:20:12.137707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.137696    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf7a0bc-5b35-406d-92ec-784510e5dede" containerName="revoke-keys"
Oct 31 04:20:12.141430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.141414    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.192903 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.192871    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"]
Oct 31 04:20:12.233568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.233517    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:20:12.233568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.233551    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:20:12.264895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.264849    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.265119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.265025    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbj5\" (UniqueName: \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.265212 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.265194    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.365938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.365899    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.366136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.365991    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.366136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.366058    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbj5\" (UniqueName: \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.366763 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.366682    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.368499 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.368475    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.419259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.419185    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbj5\" (UniqueName: \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\") pod \"maas-api-revoke-sub-77201f15-vrr8j\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.455851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.455822    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:12.553319 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.553281    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 04:20:12.668382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.668344    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 04:20:12.836797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:12.836766    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"]
Oct 31 04:20:12.838650 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:20:12.838618    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3f187f_0a0c_45e9_9e7c_3562e4f2d35f.slice/crio-e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad WatchSource:0}: Error finding container e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad: Status 404 returned error can't find the container with id e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad
Oct 31 04:20:13.052140 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:20:13.052104    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:20:13.621822 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:13.621711    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j" event={"ID":"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f","Type":"ContainerStarted","Data":"26dd5168eb9c817282a08954386c7ffe9d8d88df3f4db99b7fb51ce350bf8f79"}
Oct 31 04:20:13.622210 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:13.621823    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j" event={"ID":"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f","Type":"ContainerStarted","Data":"e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad"}
Oct 31 04:20:13.728974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:13.728918    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j" podStartSLOduration=1.7289023430000001 podStartE2EDuration="1.728902343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:13.726203182 +0000 UTC m=+4363.740764887" watchObservedRunningTime="2026-10-31 04:20:13.728902343 +0000 UTC m=+4363.743464303"
Oct 31 04:20:14.630957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:14.630925    5831 generic.go:356] "Generic (PLEG): container finished" podID="ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" containerID="26dd5168eb9c817282a08954386c7ffe9d8d88df3f4db99b7fb51ce350bf8f79" exitCode=0
Oct 31 04:20:14.631395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:14.631014    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j" event={"ID":"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f","Type":"ContainerDied","Data":"26dd5168eb9c817282a08954386c7ffe9d8d88df3f4db99b7fb51ce350bf8f79"}
Oct 31 04:20:15.797168 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.797143    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:15.865064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.865037    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\" (UniqueName: \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\") pod \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") "
Oct 31 04:20:15.865213 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.865100    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\") pod \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") "
Oct 31 04:20:15.865274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.865239    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\") pod \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\" (UID: \"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f\") "
Oct 31 04:20:15.865567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.865512    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca" pod "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" (UID: "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:15.865730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.865713    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:15.867392 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.867367    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5" pod "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" (UID: "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f"). InnerVolumeSpecName "kube-api-access-6nbj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:15.867494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.867416    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca" pod "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" (UID: "ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:15.966661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.966625    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:15.966661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:15.966655    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbj5\" (UniqueName: \"kubernetes.io/projected/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f-kube-api-access-6nbj5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:16.650013 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:16.649980    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"
Oct 31 04:20:16.650202 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:16.650011    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j" event={"ID":"ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f","Type":"ContainerDied","Data":"e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad"}
Oct 31 04:20:16.650202 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:16.650069    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0cbc10d14a01a6d92de21edde9644117c1bede291cc65cb557cf63a448817ad"
Oct 31 04:20:17.553787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:17.553739    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing" containerID="cri-o://5fe2ee7a3d955d6201a815ac83ed1f67a02e1055ae22b676fda8608ec051d07d" gracePeriod=25
Oct 31 04:20:17.670057 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:17.670002    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing" containerID="cri-o://dacc5e7e6ab084f0fcd10cb102e656c8ac56e3ccfda5e0f80093f6deeb1d94f7" gracePeriod=25
Oct 31 04:20:19.110820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.110773    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" containerID="cri-o://4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca" gracePeriod=45
Oct 31 04:20:19.149294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.149262    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 31 04:20:19.391426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.391397    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 04:20:19.505214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505177    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505415 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505276    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505415 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505316    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505415 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505357    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\" (UniqueName: \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505437    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\") pod \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\" (UID: \"ad31cd92-e0d4-4174-9f05-b2ab176b9036\") "
Oct 31 04:20:19.505632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505496    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.506103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.506017    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.506302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.506273    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.506777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.505732    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.508882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.508843    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "kube-api-access-qm7jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:19.508989 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.508919    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.508989 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.508960    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs" pod "ad31cd92-e0d4-4174-9f05-b2ab176b9036" (UID: "ad31cd92-e0d4-4174-9f05-b2ab176b9036"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:19.607054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.607021    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.607054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.607049    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7jd\" (UniqueName: \"kubernetes.io/projected/ad31cd92-e0d4-4174-9f05-b2ab176b9036-kube-api-access-qm7jd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.607054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.607060    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.607308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.607069    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad31cd92-e0d4-4174-9f05-b2ab176b9036-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.607308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.607076    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad31cd92-e0d4-4174-9f05-b2ab176b9036-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.677027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.676931    5831 generic.go:356] "Generic (PLEG): container finished" podID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerID="4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca" exitCode=0
Oct 31 04:20:19.677027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.676988    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"
Oct 31 04:20:19.677255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.677017    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" event={"ID":"ad31cd92-e0d4-4174-9f05-b2ab176b9036","Type":"ContainerDied","Data":"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"}
Oct 31 04:20:19.677255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.677093    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59" event={"ID":"ad31cd92-e0d4-4174-9f05-b2ab176b9036","Type":"ContainerDied","Data":"8bef5cae5e3e80991f92de72af1c5dcd61dbda9ee73f70913c42edb99e261d77"}
Oct 31 04:20:19.677255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.677126    5831 scope.go:122] "RemoveContainer" containerID="4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"
Oct 31 04:20:19.690616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.690593    5831 scope.go:122] "RemoveContainer" containerID="4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"
Oct 31 04:20:19.691030 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:20:19.691000    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca\": container with ID starting with 4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca not found: ID does not exist" containerID="4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"
Oct 31 04:20:19.691136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.691039    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca"} err="failed to get container status \"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca\": rpc error: code = NotFound desc = could not find container \"4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca\": container with ID starting with 4a0c588bc0e7be71a7d218de7fd9412e1522cfa51fa91525f6232d6769a6cbca not found: ID does not exist"
Oct 31 04:20:19.831239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.831204    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 04:20:19.834432 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:19.834400    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/premium-w0-7778d0-kserve-6f78554b9c-zgb59"]
Oct 31 04:20:20.136263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:20.136235    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" path="/var/lib/kubelet/pods/ad31cd92-e0d4-4174-9f05-b2ab176b9036/volumes"
Oct 31 04:20:26.963917 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:26.963872    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:20:26.964413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:26.963919    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:20:35.042605 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:35.042568    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:20:36.963542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:36.963481    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:20:36.963542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:36.963481    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:20:37.816919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.816835    5831 generic.go:356] "Generic (PLEG): container finished" podID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerID="5fe2ee7a3d955d6201a815ac83ed1f67a02e1055ae22b676fda8608ec051d07d" exitCode=0
Oct 31 04:20:37.817077 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.817004    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" event={"ID":"5e471678-5fc3-4ab8-85b5-cd34b0310337","Type":"ContainerDied","Data":"5fe2ee7a3d955d6201a815ac83ed1f67a02e1055ae22b676fda8608ec051d07d"}
Oct 31 04:20:37.819452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.819307    5831 generic.go:356] "Generic (PLEG): container finished" podID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerID="dacc5e7e6ab084f0fcd10cb102e656c8ac56e3ccfda5e0f80093f6deeb1d94f7" exitCode=0
Oct 31 04:20:37.819452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.819358    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" event={"ID":"8670eaf6-7e04-4dd1-9692-ca0795078bd2","Type":"ContainerDied","Data":"dacc5e7e6ab084f0fcd10cb102e656c8ac56e3ccfda5e0f80093f6deeb1d94f7"}
Oct 31 04:20:37.862025 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.861993    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 04:20:37.982416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:37.982375    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 04:20:38.033027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.032993    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\" (UniqueName: \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\") pod \"5e471678-5fc3-4ab8-85b5-cd34b0310337\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") "
Oct 31 04:20:38.033219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.033078    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\") pod \"5e471678-5fc3-4ab8-85b5-cd34b0310337\" (UID: \"5e471678-5fc3-4ab8-85b5-cd34b0310337\") "
Oct 31 04:20:38.033719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.033687    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume" pod "5e471678-5fc3-4ab8-85b5-cd34b0310337" (UID: "5e471678-5fc3-4ab8-85b5-cd34b0310337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:38.035364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.035344    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p" pod "5e471678-5fc3-4ab8-85b5-cd34b0310337" (UID: "5e471678-5fc3-4ab8-85b5-cd34b0310337"). InnerVolumeSpecName "kube-api-access-xvw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:38.134787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.134700    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\" (UniqueName: \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\") pod \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") "
Oct 31 04:20:38.134974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.134819    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\") pod \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\" (UID: \"8670eaf6-7e04-4dd1-9692-ca0795078bd2\") "
Oct 31 04:20:38.135266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.135240    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e471678-5fc3-4ab8-85b5-cd34b0310337-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:38.135396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.135271    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw5p\" (UniqueName: \"kubernetes.io/projected/5e471678-5fc3-4ab8-85b5-cd34b0310337-kube-api-access-xvw5p\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:38.135493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.135466    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume" pod "8670eaf6-7e04-4dd1-9692-ca0795078bd2" (UID: "8670eaf6-7e04-4dd1-9692-ca0795078bd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:38.137857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.137828    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k" pod "8670eaf6-7e04-4dd1-9692-ca0795078bd2" (UID: "8670eaf6-7e04-4dd1-9692-ca0795078bd2"). InnerVolumeSpecName "kube-api-access-v542k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:38.237118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.236864    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v542k\" (UniqueName: \"kubernetes.io/projected/8670eaf6-7e04-4dd1-9692-ca0795078bd2-kube-api-access-v542k\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:38.237118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.236898    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8670eaf6-7e04-4dd1-9692-ca0795078bd2-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:38.829332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.829252    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"
Oct 31 04:20:38.829332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.829290    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc" event={"ID":"8670eaf6-7e04-4dd1-9692-ca0795078bd2","Type":"ContainerDied","Data":"dfdbdc27f801e6b7cfbac5a46b01050a53fc64565b28ef6376b8144b144cabe9"}
Oct 31 04:20:38.829657 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.829361    5831 scope.go:122] "RemoveContainer" containerID="dacc5e7e6ab084f0fcd10cb102e656c8ac56e3ccfda5e0f80093f6deeb1d94f7"
Oct 31 04:20:38.831169 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.831142    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"
Oct 31 04:20:38.831265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.831186    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq" event={"ID":"5e471678-5fc3-4ab8-85b5-cd34b0310337","Type":"ContainerDied","Data":"5b96f7caad0f51cfa4c0f7b7218300936f403b6dc861fd126840d67dbeaa2ef7"}
Oct 31 04:20:38.840464 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.840443    5831 scope.go:122] "RemoveContainer" containerID="5fe2ee7a3d955d6201a815ac83ed1f67a02e1055ae22b676fda8608ec051d07d"
Oct 31 04:20:38.990602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:38.990569    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 04:20:39.036619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:39.036581    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7778d0-54f4b4967c-xv4zq"]
Oct 31 04:20:39.092833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:39.092732    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 04:20:39.096252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:39.096222    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7778d0-77b568fd9f-2pjlc"]
Oct 31 04:20:40.136135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:40.136100    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" path="/var/lib/kubelet/pods/5e471678-5fc3-4ab8-85b5-cd34b0310337/volumes"
Oct 31 04:20:40.136744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:20:40.136724    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" path="/var/lib/kubelet/pods/8670eaf6-7e04-4dd1-9692-ca0795078bd2/volumes"
Oct 31 04:20:42.979028 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:20:42.978986    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:21:06.437472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.437384    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"]
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.437997    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438011    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438025    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438031    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438058    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" containerName="revoke-keys"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438063    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" containerName="revoke-keys"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438072    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing"
Oct 31 04:21:06.438078 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438077    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing"
Oct 31 04:21:06.438353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438154    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad31cd92-e0d4-4174-9f05-b2ab176b9036" containerName="main"
Oct 31 04:21:06.438353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438166    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" containerName="revoke-keys"
Oct 31 04:21:06.438353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438172    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e471678-5fc3-4ab8-85b5-cd34b0310337" containerName="payload-processing"
Oct 31 04:21:06.438353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.438181    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8670eaf6-7e04-4dd1-9692-ca0795078bd2" containerName="payload-pre-processing"
Oct 31 04:21:06.441784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.441762    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.495022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.494983    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"]
Oct 31 04:21:06.536030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.536002    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:21:06.536219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.536102    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:21:06.632140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.632100    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.632330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.632147    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbkhc\" (UniqueName: \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.632330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.632310    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.733744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.733649    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.733744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.733697    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbkhc\" (UniqueName: \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.734011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.733813    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.734303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.734281    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.736248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.736230    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:06.786729 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:06.786688    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbkhc\" (UniqueName: \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\") pod \"maas-api-revoke-sub-620623ac-nlcz4\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:07.055823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:07.055745    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:07.434772 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:07.434744    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"]
Oct 31 04:21:07.436367 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:21:07.436340    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aaa020_ae00_45c0_b440_bdd3890613cc.slice/crio-d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d WatchSource:0}: Error finding container d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d: Status 404 returned error can't find the container with id d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d
Oct 31 04:21:08.060228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:08.060130    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4" event={"ID":"51aaa020-ae00-45c0-b440-bdd3890613cc","Type":"ContainerStarted","Data":"3c9fb1ecc01f121eeb9909908a13aceef54bfe777763fe8ddcef68b9ae2af142"}
Oct 31 04:21:08.060868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:08.060830    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4" event={"ID":"51aaa020-ae00-45c0-b440-bdd3890613cc","Type":"ContainerStarted","Data":"d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d"}
Oct 31 04:21:09.069357 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:09.069320    5831 generic.go:356] "Generic (PLEG): container finished" podID="51aaa020-ae00-45c0-b440-bdd3890613cc" containerID="3c9fb1ecc01f121eeb9909908a13aceef54bfe777763fe8ddcef68b9ae2af142" exitCode=0
Oct 31 04:21:09.069828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:09.069389    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4" event={"ID":"51aaa020-ae00-45c0-b440-bdd3890613cc","Type":"ContainerDied","Data":"3c9fb1ecc01f121eeb9909908a13aceef54bfe777763fe8ddcef68b9ae2af142"}
Oct 31 04:21:10.243953 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.243927    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:10.287333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.287297    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\") pod \"51aaa020-ae00-45c0-b440-bdd3890613cc\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") "
Oct 31 04:21:10.287515 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.287426    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\") pod \"51aaa020-ae00-45c0-b440-bdd3890613cc\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") "
Oct 31 04:21:10.287515 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.287477    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\" (UniqueName: \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\") pod \"51aaa020-ae00-45c0-b440-bdd3890613cc\" (UID: \"51aaa020-ae00-45c0-b440-bdd3890613cc\") "
Oct 31 04:21:10.287797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.287771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca" pod "51aaa020-ae00-45c0-b440-bdd3890613cc" (UID: "51aaa020-ae00-45c0-b440-bdd3890613cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:21:10.287969 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.287950    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:10.289745 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.289720    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc" pod "51aaa020-ae00-45c0-b440-bdd3890613cc" (UID: "51aaa020-ae00-45c0-b440-bdd3890613cc"). InnerVolumeSpecName "kube-api-access-rbkhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:10.289848 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.289758    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca" pod "51aaa020-ae00-45c0-b440-bdd3890613cc" (UID: "51aaa020-ae00-45c0-b440-bdd3890613cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:21:10.389129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.389046    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51aaa020-ae00-45c0-b440-bdd3890613cc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:10.389129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:10.389079    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbkhc\" (UniqueName: \"kubernetes.io/projected/51aaa020-ae00-45c0-b440-bdd3890613cc-kube-api-access-rbkhc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:11.086563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:11.086507    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"
Oct 31 04:21:11.086777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:11.086683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4" event={"ID":"51aaa020-ae00-45c0-b440-bdd3890613cc","Type":"ContainerDied","Data":"d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d"}
Oct 31 04:21:11.086777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:11.086735    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d20f6a06bc6b7bfda593740ef918eb71eee25f26baeb032f32160cefdb8f9e7d"
Oct 31 04:21:12.979145 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:21:12.979110    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:21:17.251661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:17.251621    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:21:41.128678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:41.128634    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:21:41.129481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:21:41.129442    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:21:42.981064 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:21:42.981021    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:22:12.982706 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:22:12.982668    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:22:37.033672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:37.033591    5831 scope.go:122] "RemoveContainer" containerID="94bd4e1ccc29d56f3a2fe1607e4ef916bbe333a9ceabdc4e4e8fb74ffa8d8c6d"
Oct 31 04:22:38.663846 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.663821    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:22:38.670002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.669979    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:22:38.674152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.674130    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:22:38.678003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.677984    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:22:38.680171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.680151    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:22:38.688187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:38.688168    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:22:42.980684 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:22:42.980642    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:22:47.697635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:47.697595    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 04:22:52.698591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:52.698550    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing" containerID="cri-o://85ae2038721d4117350afb4577d0886aba0466ba0ede3b2a8eec020e971b1656" gracePeriod=25
Oct 31 04:22:57.247419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:22:57.247372    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:23:00.663270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:00.663228    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:23:10.663730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:10.663690    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:23:13.008307 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:23:13.008181    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:23:13.090109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.090075    5831 generic.go:356] "Generic (PLEG): container finished" podID="a06384da-1a15-4600-a7df-74c26605e7ec" containerID="85ae2038721d4117350afb4577d0886aba0466ba0ede3b2a8eec020e971b1656" exitCode=0
Oct 31 04:23:13.090279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.090186    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" event={"ID":"a06384da-1a15-4600-a7df-74c26605e7ec","Type":"ContainerDied","Data":"85ae2038721d4117350afb4577d0886aba0466ba0ede3b2a8eec020e971b1656"}
Oct 31 04:23:13.090279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.090234    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc" event={"ID":"a06384da-1a15-4600-a7df-74c26605e7ec","Type":"ContainerDied","Data":"cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1"}
Oct 31 04:23:13.090279 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.090256    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb9ef8dbf60f60fe4fe61e856654e2a8d3ae2db4e8715859946006c56e0e8f1"
Oct 31 04:23:13.092663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.092640    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 04:23:13.205778 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.205719    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\" (UniqueName: \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\") pod \"a06384da-1a15-4600-a7df-74c26605e7ec\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") "
Oct 31 04:23:13.205994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.205891    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\") pod \"a06384da-1a15-4600-a7df-74c26605e7ec\" (UID: \"a06384da-1a15-4600-a7df-74c26605e7ec\") "
Oct 31 04:23:13.206281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.206255    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume" pod "a06384da-1a15-4600-a7df-74c26605e7ec" (UID: "a06384da-1a15-4600-a7df-74c26605e7ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:13.206404 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.206387    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06384da-1a15-4600-a7df-74c26605e7ec-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:13.208413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.208387    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd" pod "a06384da-1a15-4600-a7df-74c26605e7ec" (UID: "a06384da-1a15-4600-a7df-74c26605e7ec"). InnerVolumeSpecName "kube-api-access-hstwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:13.307697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:13.307605    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstwd\" (UniqueName: \"kubernetes.io/projected/a06384da-1a15-4600-a7df-74c26605e7ec-kube-api-access-hstwd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:14.099396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:14.099362    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"
Oct 31 04:23:14.251948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:14.251910    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 04:23:14.298062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:14.298027    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5b85-6f75c7b449-79chc"]
Oct 31 04:23:16.136447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:16.136408    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" path="/var/lib/kubelet/pods/a06384da-1a15-4600-a7df-74c26605e7ec/volumes"
Oct 31 04:23:34.139387 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:34.139355    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:23:37.059086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:37.059055    5831 scope.go:122] "RemoveContainer" containerID="85ae2038721d4117350afb4577d0886aba0466ba0ede3b2a8eec020e971b1656"
Oct 31 04:23:41.076294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:41.076259    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 04:23:41.076834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:41.076553    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" containerID="cri-o://f633b6610175a025e658abf5366142b384263ead006b8947095f84721ddeb2c5" gracePeriod=30
Oct 31 04:23:42.979343 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:23:42.979302    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:23:43.030973 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.030939    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:23:43.031612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.031580    5831 cpu_manager.go:414] "containerMap: removing container" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing"
Oct 31 04:23:43.031612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.031604    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing"
Oct 31 04:23:43.031784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.031686    5831 cpu_manager.go:414] "containerMap: removing container" podUID="51aaa020-ae00-45c0-b440-bdd3890613cc" containerName="revoke-keys"
Oct 31 04:23:43.031784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.031695    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51aaa020-ae00-45c0-b440-bdd3890613cc" containerName="revoke-keys"
Oct 31 04:23:43.034932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.032290    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="51aaa020-ae00-45c0-b440-bdd3890613cc" containerName="revoke-keys"
Oct 31 04:23:43.034932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.032355    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06384da-1a15-4600-a7df-74c26605e7ec" containerName="payload-processing"
Oct 31 04:23:43.039433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.039409    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.087684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.087647    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:23:43.129498 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.129455    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-26c74d32-serving-cert\""
Oct 31 04:23:43.162742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.162701    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.162947 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.162844    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.163026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.162970    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hd45\" (UniqueName: \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.163026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.162998    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.264597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.264497    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hd45\" (UniqueName: \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.264597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.264576    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.264856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.264616    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.264856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.264801    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.264981 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:23:43.264916    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-serving-cert: secret "maas-api-e2e-mt-26c74d32-serving-cert" not found
Oct 31 04:23:43.265050 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:23:43.264985    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls podName:2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01 nodeName:}" failed. No retries permitted until 2026-10-31 04:23:43.76496869 +0000 UTC m=+4573.779530379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls") pod "maas-api-e2e-mt-26c74d32-78985746c9-86nnp" (UID: "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01") : secret "maas-api-e2e-mt-26c74d32-serving-cert" not found
Oct 31 04:23:43.265284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.265265    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.267464 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.267440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.334663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.334630    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hd45\" (UniqueName: \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.773032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.772992    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.775764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.775732    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") pod \"maas-api-e2e-mt-26c74d32-78985746c9-86nnp\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:43.956767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:43.956730    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:44.125757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:44.125660    5831 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:23:44.125757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:44.125722    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:23:44.342838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:44.342804    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:23:44.344863 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:23:44.344833    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4dd43b_47a1_4ff8_b4f4_f42e62cb8d01.slice/crio-d80ba55ef1f498874c0cbeec87683c3abbca10fef9fa6a416cf1574c79d456f9 WatchSource:0}: Error finding container d80ba55ef1f498874c0cbeec87683c3abbca10fef9fa6a416cf1574c79d456f9: Status 404 returned error can't find the container with id d80ba55ef1f498874c0cbeec87683c3abbca10fef9fa6a416cf1574c79d456f9
Oct 31 04:23:44.346351 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:44.346328    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:23:45.358932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:45.358900    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" event={"ID":"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01","Type":"ContainerStarted","Data":"d80ba55ef1f498874c0cbeec87683c3abbca10fef9fa6a416cf1574c79d456f9"}
Oct 31 04:23:46.369255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.369165    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" event={"ID":"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01","Type":"ContainerStarted","Data":"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"}
Oct 31 04:23:46.369788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.369398    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:46.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.370996    5831 generic.go:356] "Generic (PLEG): container finished" podID="b143617b-97cc-4b31-be9e-725c3efb4651" containerID="f633b6610175a025e658abf5366142b384263ead006b8947095f84721ddeb2c5" exitCode=0
Oct 31 04:23:46.371147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.371073    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" event={"ID":"b143617b-97cc-4b31-be9e-725c3efb4651","Type":"ContainerDied","Data":"f633b6610175a025e658abf5366142b384263ead006b8947095f84721ddeb2c5"}
Oct 31 04:23:46.420612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.420585    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 04:23:46.477965 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.477911    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" podStartSLOduration=3.502905926 podStartE2EDuration="4.477897461s" totalImagesPullingTime="974.991535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:46.475023025 +0000 UTC m=+4576.489584734" watchObservedRunningTime="2026-10-31 04:23:46.477897461 +0000 UTC m=+4576.492459169"
Oct 31 04:23:46.506500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506568    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506616    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506666    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506697    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506794    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506806    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.506960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506837    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.506960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506949    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.507198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506986    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\") pod \"b143617b-97cc-4b31-be9e-725c3efb4651\" (UID: \"b143617b-97cc-4b31-be9e-725c3efb4651\") "
Oct 31 04:23:46.507198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507042    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.507198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.506963    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:46.507198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507136    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.507436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507234    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.507514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507469    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.507514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507489    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.507514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507504    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507518    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.507727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.507582    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b143617b-97cc-4b31-be9e-725c3efb4651-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.509461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.509434    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "kube-api-access-dvt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:46.509735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.509618    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.509982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.509961    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:23:46.509982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.509968    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token" pod "b143617b-97cc-4b31-be9e-725c3efb4651" (UID: "b143617b-97cc-4b31-be9e-725c3efb4651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:46.609133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.609091    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b143617b-97cc-4b31-be9e-725c3efb4651-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.609133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.609131    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b143617b-97cc-4b31-be9e-725c3efb4651-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.609337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.609147    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-kube-api-access-dvt66\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.609337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:46.609162    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b143617b-97cc-4b31-be9e-725c3efb4651-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:47.390732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:47.390696    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"
Oct 31 04:23:47.390732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:47.390715    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff" event={"ID":"b143617b-97cc-4b31-be9e-725c3efb4651","Type":"ContainerDied","Data":"bc22dee7de7fd3cfd3588e52c3b3820f55bb00b4ba5cc4fe813d0b9a1d0067ba"}
Oct 31 04:23:47.391443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:47.390819    5831 scope.go:122] "RemoveContainer" containerID="f633b6610175a025e658abf5366142b384263ead006b8947095f84721ddeb2c5"
Oct 31 04:23:47.553142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:47.553108    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 04:23:47.600837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:47.600798    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-jkwff"]
Oct 31 04:23:48.136911 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:48.136882    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" path="/var/lib/kubelet/pods/b143617b-97cc-4b31-be9e-725c3efb4651/volumes"
Oct 31 04:23:52.399213 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:52.399182    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:23:59.049071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:59.048986    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"]
Oct 31 04:23:59.095265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:23:59.095233    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5adf9b8d-ldn64"]
Oct 31 04:24:00.136699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:00.136662    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f157398e-7734-4d01-a21b-9a7a55e2cc85" path="/var/lib/kubelet/pods/f157398e-7734-4d01-a21b-9a7a55e2cc85/volumes"
Oct 31 04:24:05.879561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:05.879503    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:24:12.980018 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:24:12.979973    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:24:29.043236 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:29.043199    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"]
Oct 31 04:24:29.089809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:29.089773    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4deb5301-fvh8k"]
Oct 31 04:24:29.939556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:29.939054    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:24:30.138788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:24:30.138751    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb39d151-fa52-4341-9cff-d525b4301886" path="/var/lib/kubelet/pods/fb39d151-fa52-4341-9cff-d525b4301886/volumes"
Oct 31 04:24:42.978386 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:24:42.978347    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:25:04.043071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:04.043040    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"]
Oct 31 04:25:04.091483 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:04.091441    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22274710-25m96"]
Oct 31 04:25:04.134927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:04.134884    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf7a0bc-5b35-406d-92ec-784510e5dede" path="/var/lib/kubelet/pods/9bf7a0bc-5b35-406d-92ec-784510e5dede/volumes"
Oct 31 04:25:12.978749 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:25:12.978708    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:25:16.048793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:16.048751    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"]
Oct 31 04:25:16.051908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:16.051882    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77201f15-vrr8j"]
Oct 31 04:25:16.136379 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:16.136341    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f" path="/var/lib/kubelet/pods/ed3f187f-0a0c-45e9-9e7c-3562e4f2d35f/volumes"
Oct 31 04:25:26.093143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:26.093106    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 04:25:31.094165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:31.094054    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing" containerID="cri-o://8f61c1821b1c8a5e830c0683f9429dff71e52df739152d7aaea5dded13bf6e90" gracePeriod=25
Oct 31 04:25:33.907274 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:33.907227    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:25:37.105824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:37.105792    5831 scope.go:122] "RemoveContainer" containerID="aede0a8705701aade6f3c1f074a80245f057365e6c9a78ce07d15e7439b2a350"
Oct 31 04:25:37.118861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:37.118838    5831 scope.go:122] "RemoveContainer" containerID="e917113f41d0d3fc5571d21d915f1905a062ac1ef77f1e3d1960b0748237bd70"
Oct 31 04:25:42.978954 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:25:42.978913    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:25:43.907440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:43.907391    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:25:50.699330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:50.699293    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:25:51.319011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.318969    5831 generic.go:356] "Generic (PLEG): container finished" podID="3c22c6da-058e-45af-945b-519d82ea39bb" containerID="8f61c1821b1c8a5e830c0683f9429dff71e52df739152d7aaea5dded13bf6e90" exitCode=0
Oct 31 04:25:51.319214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.319034    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" event={"ID":"3c22c6da-058e-45af-945b-519d82ea39bb","Type":"ContainerDied","Data":"8f61c1821b1c8a5e830c0683f9429dff71e52df739152d7aaea5dded13bf6e90"}
Oct 31 04:25:51.352824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.352759    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 04:25:51.449777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.427226    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\" (UniqueName: \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\") pod \"3c22c6da-058e-45af-945b-519d82ea39bb\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") "
Oct 31 04:25:51.450129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.450106    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\") pod \"3c22c6da-058e-45af-945b-519d82ea39bb\" (UID: \"3c22c6da-058e-45af-945b-519d82ea39bb\") "
Oct 31 04:25:51.459997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.459913    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume" pod "3c22c6da-058e-45af-945b-519d82ea39bb" (UID: "3c22c6da-058e-45af-945b-519d82ea39bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:51.488150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.487865    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd" pod "3c22c6da-058e-45af-945b-519d82ea39bb" (UID: "3c22c6da-058e-45af-945b-519d82ea39bb"). InnerVolumeSpecName "kube-api-access-62gjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:51.565552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.552471    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62gjd\" (UniqueName: \"kubernetes.io/projected/3c22c6da-058e-45af-945b-519d82ea39bb-kube-api-access-62gjd\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:51.565552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:51.552514    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c22c6da-058e-45af-945b-519d82ea39bb-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:52.327976 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.327948    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"
Oct 31 04:25:52.328465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.327987    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx" event={"ID":"3c22c6da-058e-45af-945b-519d82ea39bb","Type":"ContainerDied","Data":"82d8519d3c26d25d5101241cd6de9fa2ced6f7bbe7f5e11203dd5f3a2488210a"}
Oct 31 04:25:52.328465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.328048    5831 scope.go:122] "RemoveContainer" containerID="8f61c1821b1c8a5e830c0683f9429dff71e52df739152d7aaea5dded13bf6e90"
Oct 31 04:25:52.486588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.486554    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 04:25:52.532487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.532457    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-861dfd-75b46f47c5-fwkbx"]
Oct 31 04:25:52.987054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.987018    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 04:25:52.987258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:52.987235    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" podUID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" containerName="maas-api" containerID="cri-o://26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216" gracePeriod=30
Oct 31 04:25:53.327940 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.327916    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 04:25:53.335263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.335235    5831 generic.go:356] "Generic (PLEG): container finished" podID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" containerID="26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216" exitCode=0
Oct 31 04:25:53.335655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.335277    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"
Oct 31 04:25:53.335655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.335327    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" event={"ID":"d48baeb3-45e6-4f02-8c97-0701d80de3ae","Type":"ContainerDied","Data":"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"}
Oct 31 04:25:53.335655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.335380    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8" event={"ID":"d48baeb3-45e6-4f02-8c97-0701d80de3ae","Type":"ContainerDied","Data":"cf6b52c2c4066c01037723975d8438bdece8742aad85418876a4a7571f9a399f"}
Oct 31 04:25:53.335655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.335411    5831 scope.go:122] "RemoveContainer" containerID="26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"
Oct 31 04:25:53.345602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.345582    5831 scope.go:122] "RemoveContainer" containerID="26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"
Oct 31 04:25:53.345888 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:25:53.345871    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216\": container with ID starting with 26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216 not found: ID does not exist" containerID="26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"
Oct 31 04:25:53.345967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.345899    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216"} err="failed to get container status \"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216\": rpc error: code = NotFound desc = could not find container \"26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216\": container with ID starting with 26d77ed6ec89963a6a736f5a97dbb688aa04554695ad19700969b8eb3cc4f216 not found: ID does not exist"
Oct 31 04:25:53.390215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.389144    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\" (UniqueName: \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\") pod \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") "
Oct 31 04:25:53.390215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.389219    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") pod \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") "
Oct 31 04:25:53.390215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.389339    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\") pod \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") "
Oct 31 04:25:53.390215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.389380    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\") pod \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\" (UID: \"d48baeb3-45e6-4f02-8c97-0701d80de3ae\") "
Oct 31 04:25:53.390979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.390884    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca" pod "d48baeb3-45e6-4f02-8c97-0701d80de3ae" (UID: "d48baeb3-45e6-4f02-8c97-0701d80de3ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:53.409629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.409579    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c" pod "d48baeb3-45e6-4f02-8c97-0701d80de3ae" (UID: "d48baeb3-45e6-4f02-8c97-0701d80de3ae"). InnerVolumeSpecName "kube-api-access-rz68c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:53.418542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.418371    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs" pod "d48baeb3-45e6-4f02-8c97-0701d80de3ae" (UID: "d48baeb3-45e6-4f02-8c97-0701d80de3ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:25:53.419789 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.419066    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls" pod "d48baeb3-45e6-4f02-8c97-0701d80de3ae" (UID: "d48baeb3-45e6-4f02-8c97-0701d80de3ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:25:53.491088 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.491054    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48baeb3-45e6-4f02-8c97-0701d80de3ae-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:53.491088 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.491087    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz68c\" (UniqueName: \"kubernetes.io/projected/d48baeb3-45e6-4f02-8c97-0701d80de3ae-kube-api-access-rz68c\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:53.491308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.491102    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:53.491308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.491115    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d48baeb3-45e6-4f02-8c97-0701d80de3ae-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:53.749822 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.749788    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 04:25:53.753100 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:53.753071    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-861dfd-d86fd86cd-th5x8"]
Oct 31 04:25:54.135369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:54.135336    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" path="/var/lib/kubelet/pods/3c22c6da-058e-45af-945b-519d82ea39bb/volumes"
Oct 31 04:25:54.135807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:25:54.135794    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" path="/var/lib/kubelet/pods/d48baeb3-45e6-4f02-8c97-0701d80de3ae/volumes"
Oct 31 04:26:10.051564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:10.051513    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"]
Oct 31 04:26:10.097268 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:10.097233    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620623ac-nlcz4"]
Oct 31 04:26:10.136051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:10.136019    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51aaa020-ae00-45c0-b440-bdd3890613cc" path="/var/lib/kubelet/pods/51aaa020-ae00-45c0-b440-bdd3890613cc/volumes"
Oct 31 04:26:12.982156 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:26:12.982120    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:26:18.733325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:18.733280    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:26:37.151372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:37.151339    5831 scope.go:122] "RemoveContainer" containerID="d1414b48785cb55f55f0c4f892f54acd9548972ae5a6c98ab8b462451453ea11"
Oct 31 04:26:37.161910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:37.161888    5831 scope.go:122] "RemoveContainer" containerID="26dd5168eb9c817282a08954386c7ffe9d8d88df3f4db99b7fb51ce350bf8f79"
Oct 31 04:26:42.979208 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:26:42.979165    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:26:49.755868 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:49.755833    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:26:52.909828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:52.909795    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 04:26:52.910268 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:52.909996    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" podUID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" containerName="maas-api" containerID="cri-o://2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d" gracePeriod=30
Oct 31 04:26:53.223205 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.223179    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 04:26:53.318561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.314459    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\") pod \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") "
Oct 31 04:26:53.379556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.334953    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca" pod "8b55214a-a131-4c5d-bfc8-aaa2553560f8" (UID: "8b55214a-a131-4c5d-bfc8-aaa2553560f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:53.379556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.371878    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") pod \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") "
Oct 31 04:26:53.379556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.372349    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\") pod \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") "
Oct 31 04:26:53.379556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.372714    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\" (UniqueName: \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\") pod \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\" (UID: \"8b55214a-a131-4c5d-bfc8-aaa2553560f8\") "
Oct 31 04:26:53.379556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.373269    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b55214a-a131-4c5d-bfc8-aaa2553560f8-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:53.394629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.391141    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls" pod "8b55214a-a131-4c5d-bfc8-aaa2553560f8" (UID: "8b55214a-a131-4c5d-bfc8-aaa2553560f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:53.394629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.393202    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs" pod "8b55214a-a131-4c5d-bfc8-aaa2553560f8" (UID: "8b55214a-a131-4c5d-bfc8-aaa2553560f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:53.394629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.393303    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb" pod "8b55214a-a131-4c5d-bfc8-aaa2553560f8" (UID: "8b55214a-a131-4c5d-bfc8-aaa2553560f8"). InnerVolumeSpecName "kube-api-access-ms5kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:53.480051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.479954    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:53.480051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.479994    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b55214a-a131-4c5d-bfc8-aaa2553560f8-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:53.480051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.480011    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5kb\" (UniqueName: \"kubernetes.io/projected/8b55214a-a131-4c5d-bfc8-aaa2553560f8-kube-api-access-ms5kb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:53.833125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.833030    5831 generic.go:356] "Generic (PLEG): container finished" podID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" containerID="2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d" exitCode=0
Oct 31 04:26:53.833125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.833080    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"
Oct 31 04:26:53.833125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.833104    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" event={"ID":"8b55214a-a131-4c5d-bfc8-aaa2553560f8","Type":"ContainerDied","Data":"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"}
Oct 31 04:26:53.833433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.833176    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf" event={"ID":"8b55214a-a131-4c5d-bfc8-aaa2553560f8","Type":"ContainerDied","Data":"9b7d4123173ef34f92b5e4ce603a457cceee4536bb7cd359161c5aedde2a2e28"}
Oct 31 04:26:53.833433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.833194    5831 scope.go:122] "RemoveContainer" containerID="2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"
Oct 31 04:26:53.852617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.852590    5831 scope.go:122] "RemoveContainer" containerID="2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"
Oct 31 04:26:53.855214 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:26:53.855182    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d\": container with ID starting with 2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d not found: ID does not exist" containerID="2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"
Oct 31 04:26:53.855355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.855225    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d"} err="failed to get container status \"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d\": rpc error: code = NotFound desc = could not find container \"2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d\": container with ID starting with 2806f5acd21e180f9bd99558f572b69807eb26f4994d7b659014446854e3960d not found: ID does not exist"
Oct 31 04:26:53.989809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.989765    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 04:26:53.994017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:53.993985    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a5f70c-5cc7976dc4-gtzsf"]
Oct 31 04:26:54.137501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:26:54.137413    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" path="/var/lib/kubelet/pods/8b55214a-a131-4c5d-bfc8-aaa2553560f8/volumes"
Oct 31 04:27:00.848140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:00.848052    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 04:27:00.848654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:00.848294    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy" containerID="cri-o://ca7f0438d95ffcad42b193a570f245cd02ec82361ba47bee62cf8512c5336598" gracePeriod=30
Oct 31 04:27:01.481572 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:01.481505    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 04:27:04.069107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.069075    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:27:04.069575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.069278    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" podUID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" containerName="maas-api" containerID="cri-o://575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99" gracePeriod=30
Oct 31 04:27:04.377402 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.377370    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:27:04.419954 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.419926    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\" (UniqueName: \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\") pod \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") "
Oct 31 04:27:04.420133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.419983    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\") pod \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") "
Oct 31 04:27:04.420133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.420090    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\") pod \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") "
Oct 31 04:27:04.420265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.420251    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") pod \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\" (UID: \"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01\") "
Oct 31 04:27:04.420496 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.420468    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca" pod "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" (UID: "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:04.420775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.420732    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:04.422429 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.422402    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs" pod "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" (UID: "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:04.422566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.422469    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45" pod "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" (UID: "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01"). InnerVolumeSpecName "kube-api-access-7hd45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:04.422795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.422777    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls" pod "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" (UID: "2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:04.522075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.522030    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hd45\" (UniqueName: \"kubernetes.io/projected/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-kube-api-access-7hd45\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:04.522075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.522077    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:04.522376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:04.522092    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:05.019952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.018834    5831 generic.go:356] "Generic (PLEG): container finished" podID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" containerID="575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99" exitCode=0
Oct 31 04:27:05.019952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.019069    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" event={"ID":"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01","Type":"ContainerDied","Data":"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"}
Oct 31 04:27:05.019952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.019121    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp" event={"ID":"2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01","Type":"ContainerDied","Data":"d80ba55ef1f498874c0cbeec87683c3abbca10fef9fa6a416cf1574c79d456f9"}
Oct 31 04:27:05.019952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.019149    5831 scope.go:122] "RemoveContainer" containerID="575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"
Oct 31 04:27:05.019952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.019296    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"
Oct 31 04:27:05.037657 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.037631    5831 scope.go:122] "RemoveContainer" containerID="575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"
Oct 31 04:27:05.038016 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:05.037990    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99\": container with ID starting with 575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99 not found: ID does not exist" containerID="575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"
Oct 31 04:27:05.038110 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.038029    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99"} err="failed to get container status \"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99\": rpc error: code = NotFound desc = could not find container \"575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99\": container with ID starting with 575e519ec856c9c4f429d31d757772f1d29e3addc3a7484c2dce7f7ce7258e99 not found: ID does not exist"
Oct 31 04:27:05.214754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.211268    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:27:05.214754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.211334    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26c74d32-78985746c9-86nnp"]
Oct 31 04:27:05.931071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.931035    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:27:05.931369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.931336    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" containerID="cri-o://1e08e898d844b7ef8cebd16cd3cd3b39c974757c9841a8e9cccce1958262a28c" gracePeriod=30
Oct 31 04:27:05.974958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.974925    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:27:05.975178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.975154    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy" containerID="cri-o://b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8" gracePeriod=30
Oct 31 04:27:05.977784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.977751    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 04:27:05.978044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.977986    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy" containerID="cri-o://62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38" gracePeriod=30
Oct 31 04:27:05.981184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.981157    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:27:05.981412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:05.981387    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy" containerID="cri-o://0ee47971753e31ecbed19e7e6f7a939278c33723e0388cce254bbdaadd2678aa" gracePeriod=30
Oct 31 04:27:06.035384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.035347    5831 generic.go:356] "Generic (PLEG): container finished" podID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerID="ca7f0438d95ffcad42b193a570f245cd02ec82361ba47bee62cf8512c5336598" exitCode=0
Oct 31 04:27:06.035723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.035435    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerDied","Data":"ca7f0438d95ffcad42b193a570f245cd02ec82361ba47bee62cf8512c5336598"}
Oct 31 04:27:06.035889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.035769    5831 scope.go:122] "RemoveContainer" containerID="2af151d3490945def1a9380bfd00fc46c874e58c4c1dea2e9775799500dfb6d4"
Oct 31 04:27:06.138167 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.138087    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" path="/var/lib/kubelet/pods/2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01/volumes"
Oct 31 04:27:06.205454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.205419    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 04:27:06.239165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239138    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239247    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239277    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239351    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239381    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239399    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239447    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239552    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239612    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.239666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239665    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.240179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239677    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:06.240179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239696    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\") pod \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\" (UID: \"3a1e9ecf-c40e-4091-9e24-4617a766b04a\") "
Oct 31 04:27:06.240179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239782    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:06.240179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.239811    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:06.240609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.240267    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.240609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.240287    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.240609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.240304    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.240609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.240318    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.240609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.240334    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.242010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.241986    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:06.242144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.242012    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "kube-api-access-nlbz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:06.242144 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.242112    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:06.242271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.242245    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo" pod "3a1e9ecf-c40e-4091-9e24-4617a766b04a" (UID: "3a1e9ecf-c40e-4091-9e24-4617a766b04a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:06.341510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.341476    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.341510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.341507    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.341510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.341517    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.341875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.341552    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbz4\" (UniqueName: \"kubernetes.io/projected/3a1e9ecf-c40e-4091-9e24-4617a766b04a-kube-api-access-nlbz4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:06.482692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:06.482645    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing" containerID="cri-o://249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d" gracePeriod=25
Oct 31 04:27:07.048325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:07.048295    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"
Oct 31 04:27:07.048542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:07.048337    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5" event={"ID":"3a1e9ecf-c40e-4091-9e24-4617a766b04a","Type":"ContainerDied","Data":"44cb9e810d39aba4f8171bde45ce57c250ac2926b75dae07478a134e5c4ecf01"}
Oct 31 04:27:07.048542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:07.048398    5831 scope.go:122] "RemoveContainer" containerID="ca7f0438d95ffcad42b193a570f245cd02ec82361ba47bee62cf8512c5336598"
Oct 31 04:27:07.200946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:07.200908    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 04:27:07.246750 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:07.246721    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a5f70c-openshift-default-56495bf787-g8zg5"]
Oct 31 04:27:08.135987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:08.135951    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" path="/var/lib/kubelet/pods/3a1e9ecf-c40e-4091-9e24-4617a766b04a/volumes"
Oct 31 04:27:09.299753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:09.299717    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:10.078417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:10.078384    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fwqzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:27:10.078637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:10.078437    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:27:11.078874 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.078837    5831 generic.go:356] "Generic (PLEG): container finished" podID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerID="0ee47971753e31ecbed19e7e6f7a939278c33723e0388cce254bbdaadd2678aa" exitCode=0
Oct 31 04:27:11.079368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.078911    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" event={"ID":"be072865-11e4-4d5f-9e2a-e21e416aea37","Type":"ContainerDied","Data":"0ee47971753e31ecbed19e7e6f7a939278c33723e0388cce254bbdaadd2678aa"}
Oct 31 04:27:11.085751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.085713    5831 generic.go:356] "Generic (PLEG): container finished" podID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerID="1e08e898d844b7ef8cebd16cd3cd3b39c974757c9841a8e9cccce1958262a28c" exitCode=0
Oct 31 04:27:11.085883 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.085854    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" event={"ID":"d92eedbd-2b47-4315-8128-4c2ddc883d00","Type":"ContainerDied","Data":"1e08e898d844b7ef8cebd16cd3cd3b39c974757c9841a8e9cccce1958262a28c"}
Oct 31 04:27:11.220151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.220123    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:27:11.312229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312182    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312230    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312260    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312291    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312315    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312344    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312622    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.312945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312675    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.312945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.312731    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.313403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.313378    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\") pod \"be072865-11e4-4d5f-9e2a-e21e416aea37\" (UID: \"be072865-11e4-4d5f-9e2a-e21e416aea37\") "
Oct 31 04:27:11.313510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.313488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.313751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.313715    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.314059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314036    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.315291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314232    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.315291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314289    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.315291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314304    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.315291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314318    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.315291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.314395    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.315842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.315809    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.317166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.317142    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.317941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.317920    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "kube-api-access-bqspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.318952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.318935    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token" pod "be072865-11e4-4d5f-9e2a-e21e416aea37" (UID: "be072865-11e4-4d5f-9e2a-e21e416aea37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.337439 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.337411    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:27:11.340365 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.340341    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 04:27:11.345695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.345672    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:27:11.415205 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415169    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.415378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415222    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415256    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.415378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415290    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415330    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415361    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415387    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415437    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415464    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415491    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415488    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415542    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415570    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415607    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.415710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415636    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415840    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.416194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415993    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.416194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416122    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.416360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.415713    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.416360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416282    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416377    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.416479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416413    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.416479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416441    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416479 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416470    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416501    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.416714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416576    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416615    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.416714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416652    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.416714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416700    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\") pod \"d92eedbd-2b47-4315-8128-4c2ddc883d00\" (UID: \"d92eedbd-2b47-4315-8128-4c2ddc883d00\") "
Oct 31 04:27:11.416993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416739    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\") pod \"fca94099-d551-4e53-a9aa-785372780924\" (UID: \"fca94099-d551-4e53-a9aa-785372780924\") "
Oct 31 04:27:11.416993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.416769    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\") pod \"1123a65b-1994-4d76-82c5-0fe4836596f0\" (UID: \"1123a65b-1994-4d76-82c5-0fe4836596f0\") "
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.417894    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.418446    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.418765    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.418923    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.419066    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.419517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.419259    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:11.423239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.423216    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.423708 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.423682    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.425757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.423907    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.425757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.424098    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.425757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.425388    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.426913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.426282    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.426913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.426509    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.431187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431163    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "kube-api-access-mnc6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.431711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431516    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.431818 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431759    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqspm\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-kube-api-access-bqspm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431818 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431785    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431803    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431835    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431849    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431863    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431876    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be072865-11e4-4d5f-9e2a-e21e416aea37-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431890    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92eedbd-2b47-4315-8128-4c2ddc883d00-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431905    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431921    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431936    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431950    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.431967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431963    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be072865-11e4-4d5f-9e2a-e21e416aea37-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431976    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1123a65b-1994-4d76-82c5-0fe4836596f0-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.431990    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca94099-d551-4e53-a9aa-785372780924-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432004    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432018    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432032    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432058    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432072    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.432497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.432086    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.433043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.433022    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.435266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435237    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "kube-api-access-4x2lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.435798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435759    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.435798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435762    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy" pod "d92eedbd-2b47-4315-8128-4c2ddc883d00" (UID: "d92eedbd-2b47-4315-8128-4c2ddc883d00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:11.435943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435803    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.435943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435928    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "kube-api-access-drhv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:11.436026 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.435935    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo" pod "1123a65b-1994-4d76-82c5-0fe4836596f0" (UID: "1123a65b-1994-4d76-82c5-0fe4836596f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.437703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.437681    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo" pod "fca94099-d551-4e53-a9aa-785372780924" (UID: "fca94099-d551-4e53-a9aa-785372780924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:11.532831 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532760    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.532831 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532792    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca94099-d551-4e53-a9aa-785372780924-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.532831 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532810    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnc6c\" (UniqueName: \"kubernetes.io/projected/d92eedbd-2b47-4315-8128-4c2ddc883d00-kube-api-access-mnc6c\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532853    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532868    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2lc\" (UniqueName: \"kubernetes.io/projected/fca94099-d551-4e53-a9aa-785372780924-kube-api-access-4x2lc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532883    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532897    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drhv2\" (UniqueName: \"kubernetes.io/projected/1123a65b-1994-4d76-82c5-0fe4836596f0-kube-api-access-drhv2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532912    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92eedbd-2b47-4315-8128-4c2ddc883d00-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532927    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1123a65b-1994-4d76-82c5-0fe4836596f0-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532940    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:11.533058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:11.532954    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca94099-d551-4e53-a9aa-785372780924-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:12.094909 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.094870    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt" event={"ID":"be072865-11e4-4d5f-9e2a-e21e416aea37","Type":"ContainerDied","Data":"07180e130675e890fb19d6a5971d32f9bff20028afb1b059c69d91241a01bf50"}
Oct 31 04:27:12.095398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.094946    5831 scope.go:122] "RemoveContainer" containerID="0ee47971753e31ecbed19e7e6f7a939278c33723e0388cce254bbdaadd2678aa"
Oct 31 04:27:12.095398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.094885    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"
Oct 31 04:27:12.096663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.096499    5831 generic.go:356] "Generic (PLEG): container finished" podID="fca94099-d551-4e53-a9aa-785372780924" containerID="62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38" exitCode=0
Oct 31 04:27:12.096663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.096573    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"
Oct 31 04:27:12.096663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.096629    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" event={"ID":"fca94099-d551-4e53-a9aa-785372780924","Type":"ContainerDied","Data":"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"}
Oct 31 04:27:12.096663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.096666    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62" event={"ID":"fca94099-d551-4e53-a9aa-785372780924","Type":"ContainerDied","Data":"63174388b4b6953557f9dfbdb4d063ba063f496e055e4f0ce0f6ba3e45637417"}
Oct 31 04:27:12.098888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.098864    5831 generic.go:356] "Generic (PLEG): container finished" podID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerID="b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8" exitCode=0
Oct 31 04:27:12.099066 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.099050    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" event={"ID":"1123a65b-1994-4d76-82c5-0fe4836596f0","Type":"ContainerDied","Data":"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"}
Oct 31 04:27:12.099119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.099097    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2" event={"ID":"1123a65b-1994-4d76-82c5-0fe4836596f0","Type":"ContainerDied","Data":"273d230411da407f8e3cf05be02e9dc892faff9eab38387e15be30e81559c5a0"}
Oct 31 04:27:12.099175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.099163    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"
Oct 31 04:27:12.114165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.114140    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv" event={"ID":"d92eedbd-2b47-4315-8128-4c2ddc883d00","Type":"ContainerDied","Data":"d7bb6589051df5aa539c1d9184df9ce5652f5f6c1af0cd5c8d2c2946dc9c746d"}
Oct 31 04:27:12.114294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.114217    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"
Oct 31 04:27:12.114938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.114743    5831 scope.go:122] "RemoveContainer" containerID="62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"
Oct 31 04:27:12.125389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.125369    5831 scope.go:122] "RemoveContainer" containerID="62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"
Oct 31 04:27:12.125745 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:12.125716    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38\": container with ID starting with 62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38 not found: ID does not exist" containerID="62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"
Oct 31 04:27:12.125823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.125753    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38"} err="failed to get container status \"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38\": rpc error: code = NotFound desc = could not find container \"62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38\": container with ID starting with 62ce81ac3501bd54673ad1c1c1d91f9cc3cec27879ab588e379b2731acdf3b38 not found: ID does not exist"
Oct 31 04:27:12.125823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.125771    5831 scope.go:122] "RemoveContainer" containerID="b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"
Oct 31 04:27:12.135777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.135761    5831 scope.go:122] "RemoveContainer" containerID="b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"
Oct 31 04:27:12.136012 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:12.135992    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8\": container with ID starting with b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8 not found: ID does not exist" containerID="b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"
Oct 31 04:27:12.136094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.136023    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8"} err="failed to get container status \"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8\": rpc error: code = NotFound desc = could not find container \"b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8\": container with ID starting with b104c3854c38687b39b30ae1520f15e12092de84664b949b8c128ba2feedffc8 not found: ID does not exist"
Oct 31 04:27:12.136094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.136047    5831 scope.go:122] "RemoveContainer" containerID="1e08e898d844b7ef8cebd16cd3cd3b39c974757c9841a8e9cccce1958262a28c"
Oct 31 04:27:12.248713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.248631    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:27:12.254490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.254459    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fwqzv"]
Oct 31 04:27:12.367295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.367254    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:27:12.370311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.370284    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jlkc2"]
Oct 31 04:27:12.458728 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.458685    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 04:27:12.464768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.464739    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hxx62"]
Oct 31 04:27:12.575584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.575478    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:27:12.588975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:12.588943    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qcfrt"]
Oct 31 04:27:12.978955 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:12.978918    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:27:14.134793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:14.134762    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" path="/var/lib/kubelet/pods/1123a65b-1994-4d76-82c5-0fe4836596f0/volumes"
Oct 31 04:27:14.135362 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:14.135347    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" path="/var/lib/kubelet/pods/be072865-11e4-4d5f-9e2a-e21e416aea37/volumes"
Oct 31 04:27:14.135780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:14.135767    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" path="/var/lib/kubelet/pods/d92eedbd-2b47-4315-8128-4c2ddc883d00/volumes"
Oct 31 04:27:14.136156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:14.136144    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca94099-d551-4e53-a9aa-785372780924" path="/var/lib/kubelet/pods/fca94099-d551-4e53-a9aa-785372780924/volumes"
Oct 31 04:27:19.299783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:19.299735    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:19.358349 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:19.358312    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:27:21.467021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:21.466983    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 04:27:21.637124 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:21.637087    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 04:27:22.344916 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.344878    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"]
Oct 31 04:27:22.345381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345368    5831 cpu_manager.go:414] "containerMap: removing container" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345382    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345392    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345397    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345413    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345419    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345426    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:27:22.345433 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345431    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345437    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345442    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345448    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345452    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345463    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345467    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345477    5831 cpu_manager.go:414] "containerMap: removing container" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345482    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345494    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345499    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345505    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345509    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345610    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="be072865-11e4-4d5f-9e2a-e21e416aea37" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345619    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c22c6da-058e-45af-945b-519d82ea39bb" containerName="payload-pre-processing"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345627    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b143617b-97cc-4b31-be9e-725c3efb4651" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345633    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345640    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b55214a-a131-4c5d-bfc8-aaa2553560f8" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345646    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4dd43b-47a1-4ff8-b4f4-f42e62cb8d01" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345653    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48baeb3-45e6-4f02-8c97-0701d80de3ae" containerName="maas-api"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345661    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca94099-d551-4e53-a9aa-785372780924" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345667    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345672    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92eedbd-2b47-4315-8128-4c2ddc883d00" containerName="istio-proxy"
Oct 31 04:27:22.345691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.345677    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1123a65b-1994-4d76-82c5-0fe4836596f0" containerName="istio-proxy"
Oct 31 04:27:22.349913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.349892    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.354828 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.354793    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.354974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.354958    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.355048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.355005    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5rg\" (UniqueName: \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.400503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.400337    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:27:22.400503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.400337    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:27:22.404263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.404230    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"]
Oct 31 04:27:22.456184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.456147    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.456376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.456201    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5rg\" (UniqueName: \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.456376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.456275    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.457020 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.456998    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.458902 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.458880    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.511159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.511120    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5rg\" (UniqueName: \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\") pod \"maas-api-revoke-sub-2a715685-l2rbr\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:22.664104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:22.664033    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:23.039119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:23.039094    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"]
Oct 31 04:27:23.042690 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:27:23.042658    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0c2efa_08be_43b8_a55e_e371e8a3ed16.slice/crio-0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453 WatchSource:0}: Error finding container 0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453: Status 404 returned error can't find the container with id 0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453
Oct 31 04:27:23.198904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:23.198862    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr" event={"ID":"2d0c2efa-08be-43b8-a55e-e371e8a3ed16","Type":"ContainerStarted","Data":"0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453"}
Oct 31 04:27:24.211413 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:24.211314    5831 generic.go:356] "Generic (PLEG): container finished" podID="2d0c2efa-08be-43b8-a55e-e371e8a3ed16" containerID="1c901d8ba3b2f0fab409f51d6b32c50c4ae4a0153af682acf196eebf0a6fb4b0" exitCode=0
Oct 31 04:27:24.212292 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:24.212257    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr" event={"ID":"2d0c2efa-08be-43b8-a55e-e371e8a3ed16","Type":"ContainerDied","Data":"1c901d8ba3b2f0fab409f51d6b32c50c4ae4a0153af682acf196eebf0a6fb4b0"}
Oct 31 04:27:25.375522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.375496    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:25.502482 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.502391    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\") pod \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") "
Oct 31 04:27:25.502694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.502481    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\" (UniqueName: \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\") pod \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") "
Oct 31 04:27:25.502694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.502511    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\") pod \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\" (UID: \"2d0c2efa-08be-43b8-a55e-e371e8a3ed16\") "
Oct 31 04:27:25.502844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.502771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca" pod "2d0c2efa-08be-43b8-a55e-e371e8a3ed16" (UID: "2d0c2efa-08be-43b8-a55e-e371e8a3ed16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:25.503011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.502984    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:25.504792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.504767    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg" pod "2d0c2efa-08be-43b8-a55e-e371e8a3ed16" (UID: "2d0c2efa-08be-43b8-a55e-e371e8a3ed16"). InnerVolumeSpecName "kube-api-access-qz5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:25.505287 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.505266    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca" pod "2d0c2efa-08be-43b8-a55e-e371e8a3ed16" (UID: "2d0c2efa-08be-43b8-a55e-e371e8a3ed16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:25.604692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.604646    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5rg\" (UniqueName: \"kubernetes.io/projected/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-kube-api-access-qz5rg\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:25.604692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:25.604693    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0c2efa-08be-43b8-a55e-e371e8a3ed16-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.226565 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.226515    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr" event={"ID":"2d0c2efa-08be-43b8-a55e-e371e8a3ed16","Type":"ContainerDied","Data":"0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453"}
Oct 31 04:27:26.226565 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.226552    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"
Oct 31 04:27:26.226801 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.226589    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cac15d048a39d59072c57b1f6dada1967bd8d01f93c8fd61895b93ea4707453"
Oct 31 04:27:26.467867 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.467821    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" podUID="8d16824e-2576-4aa9-9d2b-8ff73343732c" containerName="payload-processing" containerID="cri-o://84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705" gracePeriod=25
Oct 31 04:27:26.637653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.637601    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" podUID="92577884-e162-433f-ab59-50b7ff56a1af" containerName="payload-pre-processing" containerID="cri-o://cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc" gracePeriod=25
Oct 31 04:27:26.804260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.804231    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 04:27:26.808827 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.808805    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 04:27:26.820514 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.820488    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\" (UniqueName: \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\") pod \"8d16824e-2576-4aa9-9d2b-8ff73343732c\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") "
Oct 31 04:27:26.820675 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.820561    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\" (UniqueName: \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\") pod \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") "
Oct 31 04:27:26.820728 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.820699    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\") pod \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\" (UID: \"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872\") "
Oct 31 04:27:26.820789 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.820747    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\") pod \"8d16824e-2576-4aa9-9d2b-8ff73343732c\" (UID: \"8d16824e-2576-4aa9-9d2b-8ff73343732c\") "
Oct 31 04:27:26.821500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.821104    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume" pod "9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" (UID: "9f0aad2b-b7a5-44c5-9e2e-33f6edf64872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:26.821500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.821161    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume" pod "8d16824e-2576-4aa9-9d2b-8ff73343732c" (UID: "8d16824e-2576-4aa9-9d2b-8ff73343732c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:26.823575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.823513    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr" pod "8d16824e-2576-4aa9-9d2b-8ff73343732c" (UID: "8d16824e-2576-4aa9-9d2b-8ff73343732c"). InnerVolumeSpecName "kube-api-access-jc6mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:26.824847 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.824820    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6" pod "9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" (UID: "9f0aad2b-b7a5-44c5-9e2e-33f6edf64872"). InnerVolumeSpecName "kube-api-access-xshr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:26.900622 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.900590    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.921803    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\") pod \"92577884-e162-433f-ab59-50b7ff56a1af\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") "
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.921928    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\" (UniqueName: \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\") pod \"92577884-e162-433f-ab59-50b7ff56a1af\" (UID: \"92577884-e162-433f-ab59-50b7ff56a1af\") "
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922174    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume" pod "92577884-e162-433f-ab59-50b7ff56a1af" (UID: "92577884-e162-433f-ab59-50b7ff56a1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922382    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6mr\" (UniqueName: \"kubernetes.io/projected/8d16824e-2576-4aa9-9d2b-8ff73343732c-kube-api-access-jc6mr\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922407    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshr6\" (UniqueName: \"kubernetes.io/projected/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-kube-api-access-xshr6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922423    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922441    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d16824e-2576-4aa9-9d2b-8ff73343732c-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.923578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.922454    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92577884-e162-433f-ab59-50b7ff56a1af-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.924886 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:26.924851    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc" pod "92577884-e162-433f-ab59-50b7ff56a1af" (UID: "92577884-e162-433f-ab59-50b7ff56a1af"). InnerVolumeSpecName "kube-api-access-trsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:27.023809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.023723    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trsvc\" (UniqueName: \"kubernetes.io/projected/92577884-e162-433f-ab59-50b7ff56a1af-kube-api-access-trsvc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:27.234606 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.234569    5831 generic.go:356] "Generic (PLEG): container finished" podID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerID="249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d" exitCode=0
Oct 31 04:27:27.234819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.234617    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"
Oct 31 04:27:27.234819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.234669    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" event={"ID":"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872","Type":"ContainerDied","Data":"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"}
Oct 31 04:27:27.234819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.234728    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9" event={"ID":"9f0aad2b-b7a5-44c5-9e2e-33f6edf64872","Type":"ContainerDied","Data":"659d87a723180c0f81d3c93f9b59ab1c2c5a363c835eeaddfbfb3ddf4a502e01"}
Oct 31 04:27:27.234819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.234757    5831 scope.go:122] "RemoveContainer" containerID="249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"
Oct 31 04:27:27.236294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.236262    5831 generic.go:356] "Generic (PLEG): container finished" podID="8d16824e-2576-4aa9-9d2b-8ff73343732c" containerID="84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705" exitCode=0
Oct 31 04:27:27.236613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.236354    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" event={"ID":"8d16824e-2576-4aa9-9d2b-8ff73343732c","Type":"ContainerDied","Data":"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"}
Oct 31 04:27:27.236613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.236392    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p" event={"ID":"8d16824e-2576-4aa9-9d2b-8ff73343732c","Type":"ContainerDied","Data":"142cf95c66765f33a92bd28e5be5c39897feaf401ebd2b48cf5856340f90ecc5"}
Oct 31 04:27:27.236613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.236448    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"
Oct 31 04:27:27.247048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.243407    5831 generic.go:356] "Generic (PLEG): container finished" podID="92577884-e162-433f-ab59-50b7ff56a1af" containerID="cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc" exitCode=0
Oct 31 04:27:27.247048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.243455    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"
Oct 31 04:27:27.247048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.243499    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" event={"ID":"92577884-e162-433f-ab59-50b7ff56a1af","Type":"ContainerDied","Data":"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"}
Oct 31 04:27:27.247048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.243572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m" event={"ID":"92577884-e162-433f-ab59-50b7ff56a1af","Type":"ContainerDied","Data":"7cdd0916a6af0102f8aafd61980836953b214b92c3ad14551725f0116033d029"}
Oct 31 04:27:27.247580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.247114    5831 scope.go:122] "RemoveContainer" containerID="249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"
Oct 31 04:27:27.247580 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:27.247454    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d\": container with ID starting with 249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d not found: ID does not exist" containerID="249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"
Oct 31 04:27:27.247580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.247491    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d"} err="failed to get container status \"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d\": rpc error: code = NotFound desc = could not find container \"249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d\": container with ID starting with 249e3ddef619dbd8dfd7bb7d9df715b9a7c35dc067bbac11f29b60109f2e4f9d not found: ID does not exist"
Oct 31 04:27:27.247580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.247514    5831 scope.go:122] "RemoveContainer" containerID="84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"
Oct 31 04:27:27.261004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.260989    5831 scope.go:122] "RemoveContainer" containerID="84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"
Oct 31 04:27:27.261386 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:27.261365    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705\": container with ID starting with 84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705 not found: ID does not exist" containerID="84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"
Oct 31 04:27:27.261489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.261396    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705"} err="failed to get container status \"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705\": rpc error: code = NotFound desc = could not find container \"84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705\": container with ID starting with 84165a79fb9a57efa40a6f8d6c17b5caa9fb5726a6209605b1ccf6bf215f6705 not found: ID does not exist"
Oct 31 04:27:27.261489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.261420    5831 scope.go:122] "RemoveContainer" containerID="cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"
Oct 31 04:27:27.271032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.271014    5831 scope.go:122] "RemoveContainer" containerID="cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"
Oct 31 04:27:27.271287 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:27.271270    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc\": container with ID starting with cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc not found: ID does not exist" containerID="cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"
Oct 31 04:27:27.271348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.271296    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc"} err="failed to get container status \"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc\": rpc error: code = NotFound desc = could not find container \"cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc\": container with ID starting with cb101152e9fd608a5d197fa37595bd09f2ce6ecdd6976508005fde77bb12dadc not found: ID does not exist"
Oct 31 04:27:27.387408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.387321    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 04:27:27.393865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.393826    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2339ed-786bf5c4f7-9htx9"]
Oct 31 04:27:27.494955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.494913    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 04:27:27.498943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.498917    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d39257-6847b799d7-rfs7m"]
Oct 31 04:27:27.604015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.602229    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 04:27:27.610766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:27.610734    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d39257-6fbd4997d5-bgg6p"]
Oct 31 04:27:28.135660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:28.135627    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d16824e-2576-4aa9-9d2b-8ff73343732c" path="/var/lib/kubelet/pods/8d16824e-2576-4aa9-9d2b-8ff73343732c/volumes"
Oct 31 04:27:28.136051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:28.136038    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92577884-e162-433f-ab59-50b7ff56a1af" path="/var/lib/kubelet/pods/92577884-e162-433f-ab59-50b7ff56a1af/volumes"
Oct 31 04:27:28.136375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:28.136364    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" path="/var/lib/kubelet/pods/9f0aad2b-b7a5-44c5-9e2e-33f6edf64872/volumes"
Oct 31 04:27:36.667510 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:36.667471    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 04:27:36.668019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:36.667736    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" podUID="b962db6c-ba7e-4f3e-91a7-76187df94545" containerName="maas-api" containerID="cri-o://5842ea8de9f15946947b09d236cb61ef131bade4d2465a1f976536a2a1751d89" gracePeriod=30
Oct 31 04:27:36.960834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:36.960798    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 04:27:37.040155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040121    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\") pod \"b962db6c-ba7e-4f3e-91a7-76187df94545\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") "
Oct 31 04:27:37.040337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040166    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\") pod \"b962db6c-ba7e-4f3e-91a7-76187df94545\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") "
Oct 31 04:27:37.040337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040229    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\" (UniqueName: \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\") pod \"b962db6c-ba7e-4f3e-91a7-76187df94545\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") "
Oct 31 04:27:37.040434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040373    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") pod \"b962db6c-ba7e-4f3e-91a7-76187df94545\" (UID: \"b962db6c-ba7e-4f3e-91a7-76187df94545\") "
Oct 31 04:27:37.040647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040609    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca" pod "b962db6c-ba7e-4f3e-91a7-76187df94545" (UID: "b962db6c-ba7e-4f3e-91a7-76187df94545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:37.040824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.040775    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b962db6c-ba7e-4f3e-91a7-76187df94545-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:37.042754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.042711    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs" pod "b962db6c-ba7e-4f3e-91a7-76187df94545" (UID: "b962db6c-ba7e-4f3e-91a7-76187df94545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:37.042894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.042824    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls" pod "b962db6c-ba7e-4f3e-91a7-76187df94545" (UID: "b962db6c-ba7e-4f3e-91a7-76187df94545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:37.043267 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.043231    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b" pod "b962db6c-ba7e-4f3e-91a7-76187df94545" (UID: "b962db6c-ba7e-4f3e-91a7-76187df94545"). InnerVolumeSpecName "kube-api-access-rbs4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:37.142463 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.142417    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:37.142666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.142542    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b962db6c-ba7e-4f3e-91a7-76187df94545-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:37.142666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.142563    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbs4b\" (UniqueName: \"kubernetes.io/projected/b962db6c-ba7e-4f3e-91a7-76187df94545-kube-api-access-rbs4b\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:37.206424 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.206345    5831 scope.go:122] "RemoveContainer" containerID="5842ea8de9f15946947b09d236cb61ef131bade4d2465a1f976536a2a1751d89"
Oct 31 04:27:37.217081 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.217056    5831 scope.go:122] "RemoveContainer" containerID="3c9fb1ecc01f121eeb9909908a13aceef54bfe777763fe8ddcef68b9ae2af142"
Oct 31 04:27:37.320759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.320719    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" event={"ID":"b962db6c-ba7e-4f3e-91a7-76187df94545","Type":"ContainerDied","Data":"5842ea8de9f15946947b09d236cb61ef131bade4d2465a1f976536a2a1751d89"}
Oct 31 04:27:37.320936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.320787    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5" event={"ID":"b962db6c-ba7e-4f3e-91a7-76187df94545","Type":"ContainerDied","Data":"bd9f102e3267d0c6418c9cc5cce33f160f2e842c4526dd4f11030725675c1611"}
Oct 31 04:27:37.320936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.320810    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"
Oct 31 04:27:37.535268 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.533306    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 04:27:37.540699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:37.540667    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-93f411-754f9555f4-nnvx5"]
Oct 31 04:27:38.136772 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.136741    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b962db6c-ba7e-4f3e-91a7-76187df94545" path="/var/lib/kubelet/pods/b962db6c-ba7e-4f3e-91a7-76187df94545/volumes"
Oct 31 04:27:38.716039 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.716007    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:27:38.722749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.722726    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:27:38.726056 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.726037    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:27:38.729398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.729379    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:27:38.732319 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.732303    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:27:38.735666 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:38.735648    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:27:42.979984 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:27:42.979946    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:27:50.091897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:50.091861    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:27:52.416871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:52.416834    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 04:27:52.524501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:52.524469    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 04:27:54.856254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:54.856222    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 04:27:54.963638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:54.963603    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 04:27:57.418129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:57.418089    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing" containerID="cri-o://ba2ba04d0db412d5745b9fd4437ba48a6b95d0116cb6dd1b28f6138d0acdf4e6" gracePeriod=25
Oct 31 04:27:57.470378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:57.470339    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:57.525071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:57.525014    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing" containerID="cri-o://d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4" gracePeriod=25
Oct 31 04:27:59.856872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:59.856805    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing" containerID="cri-o://4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab" gracePeriod=25
Oct 31 04:27:59.964914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:27:59.964872    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing" containerID="cri-o://f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0" gracePeriod=25
Oct 31 04:28:03.195692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:03.195650    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 04:28:03.196219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:03.195952    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy" containerID="cri-o://44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759" gracePeriod=30
Oct 31 04:28:03.530792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:03.530701    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:03.530792 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:03.530760    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:07.469854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:07.469811    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:07.470260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:07.469820    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:08.465954 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.465879    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 04:28:08.533875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.533838    5831 generic.go:356] "Generic (PLEG): container finished" podID="feb36c39-f27d-4902-b7da-184bf3641495" containerID="44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759" exitCode=0
Oct 31 04:28:08.534353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.533902    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"
Oct 31 04:28:08.534353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.533927    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" event={"ID":"feb36c39-f27d-4902-b7da-184bf3641495","Type":"ContainerDied","Data":"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"}
Oct 31 04:28:08.534353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.533979    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2" event={"ID":"feb36c39-f27d-4902-b7da-184bf3641495","Type":"ContainerDied","Data":"4af15c18e77f70d7ba8d1f5f38c4f7c03eebac48a74f7f291f1c3e2f136c644a"}
Oct 31 04:28:08.534353 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.534004    5831 scope.go:122] "RemoveContainer" containerID="44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"
Oct 31 04:28:08.544442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.544425    5831 scope.go:122] "RemoveContainer" containerID="44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"
Oct 31 04:28:08.544737 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:08.544719    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759\": container with ID starting with 44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759 not found: ID does not exist" containerID="44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"
Oct 31 04:28:08.544801 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.544747    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759"} err="failed to get container status \"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759\": rpc error: code = NotFound desc = could not find container \"44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759\": container with ID starting with 44a52c82a8fdd6dad7adb5faded5d3fce626fdc6c90cbed40191b7e8ac1e0759 not found: ID does not exist"
Oct 31 04:28:08.595713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595675    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.595908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595782    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.595908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595815    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.595908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595850    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.595908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595888    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.596147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595921    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.596147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.595992    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.596147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596020    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.596147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596067    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\") pod \"feb36c39-f27d-4902-b7da-184bf3641495\" (UID: \"feb36c39-f27d-4902-b7da-184bf3641495\") "
Oct 31 04:28:08.596405 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596380    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:08.596569 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596544    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb36c39-f27d-4902-b7da-184bf3641495-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.596644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596583    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:08.596816 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596792    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:08.596933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596911    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:08.597013 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.596942    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:08.598456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.598430    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:08.598759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.598739    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:28:08.598997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.598967    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:08.599225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.599205    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh" pod "feb36c39-f27d-4902-b7da-184bf3641495" (UID: "feb36c39-f27d-4902-b7da-184bf3641495"). InnerVolumeSpecName "kube-api-access-crmdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:08.697856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697804    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb36c39-f27d-4902-b7da-184bf3641495-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.697856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697857    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697872    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697885    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmdh\" (UniqueName: \"kubernetes.io/projected/feb36c39-f27d-4902-b7da-184bf3641495-kube-api-access-crmdh\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697901    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697914    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697928    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.698129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.697942    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb36c39-f27d-4902-b7da-184bf3641495-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:08.947375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.947342    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 04:28:08.950941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:08.950916    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-zx2z2"]
Oct 31 04:28:10.134954 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:10.134920    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb36c39-f27d-4902-b7da-184bf3641495" path="/var/lib/kubelet/pods/feb36c39-f27d-4902-b7da-184bf3641495/volumes"
Oct 31 04:28:12.977690 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:12.977653    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:28:13.530814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:13.530776    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:13.530814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:13.530809    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:15.284659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:15.284627    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 04:28:15.285151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:15.284842    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy" containerID="cri-o://c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101" gracePeriod=30
Oct 31 04:28:16.369797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.369748    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"]
Oct 31 04:28:16.370447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370429    5831 cpu_manager.go:414] "containerMap: removing container" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy"
Oct 31 04:28:16.370512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370450    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy"
Oct 31 04:28:16.370512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370480    5831 cpu_manager.go:414] "containerMap: removing container" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing"
Oct 31 04:28:16.370512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370488    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing"
Oct 31 04:28:16.370512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370509    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8d16824e-2576-4aa9-9d2b-8ff73343732c" containerName="payload-processing"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370516    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d16824e-2576-4aa9-9d2b-8ff73343732c" containerName="payload-processing"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370565    5831 cpu_manager.go:414] "containerMap: removing container" podUID="92577884-e162-433f-ab59-50b7ff56a1af" containerName="payload-pre-processing"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370572    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92577884-e162-433f-ab59-50b7ff56a1af" containerName="payload-pre-processing"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370583    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2d0c2efa-08be-43b8-a55e-e371e8a3ed16" containerName="revoke-keys"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370591    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0c2efa-08be-43b8-a55e-e371e8a3ed16" containerName="revoke-keys"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370608    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370614    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e9ecf-c40e-4091-9e24-4617a766b04a" containerName="istio-proxy"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370643    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b962db6c-ba7e-4f3e-91a7-76187df94545" containerName="maas-api"
Oct 31 04:28:16.370686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370650    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962db6c-ba7e-4f3e-91a7-76187df94545" containerName="maas-api"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370761    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962db6c-ba7e-4f3e-91a7-76187df94545" containerName="maas-api"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370773    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb36c39-f27d-4902-b7da-184bf3641495" containerName="istio-proxy"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370783    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0aad2b-b7a5-44c5-9e2e-33f6edf64872" containerName="payload-processing"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370792    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d16824e-2576-4aa9-9d2b-8ff73343732c" containerName="payload-processing"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370804    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="92577884-e162-433f-ab59-50b7ff56a1af" containerName="payload-pre-processing"
Oct 31 04:28:16.371023 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.370814    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0c2efa-08be-43b8-a55e-e371e8a3ed16" containerName="revoke-keys"
Oct 31 04:28:16.376027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.376004    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.425505 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.425473    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"]
Oct 31 04:28:16.468265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.468208    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:28:16.468428 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.468277    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:28:16.469885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.469853    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:28:16.469885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.469859    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:28:16.470069 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.469951    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.470069 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.470013    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.470176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.470150    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcrx\" (UniqueName: \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.570774 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.570740    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.570987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.570827    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcrx\" (UniqueName: \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.570987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.570887    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.571434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.571405    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.573677 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.573653    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.624282 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.624193    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcrx\" (UniqueName: \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\") pod \"maas-api-revoke-sub-37a390f2-fwjn9\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:16.689273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:16.689237    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:17.058272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.058243    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"]
Oct 31 04:28:17.061294 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:17.061258    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5fe49c_d378_4dba_a5d1_1f1d0be5df03.slice/crio-51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72 WatchSource:0}: Error finding container 51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72: Status 404 returned error can't find the container with id 51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72
Oct 31 04:28:17.468996 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:17.468969    5831 logging.go:55] [core] [Channel #18480 SubChannel #18482]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.33:50052", ServerName: "10.132.1.33:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.33:50052: connect: connection refused"
Oct 31 04:28:17.469669 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.469644    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:17.592305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.592269    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9" event={"ID":"de5fe49c-d378-4dba-a5d1-1f1d0be5df03","Type":"ContainerStarted","Data":"e2d5c7b051e536efc82895ca5d079e98749bd64d4d7032656232666b866b8aec"}
Oct 31 04:28:17.592555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.592405    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9" event={"ID":"de5fe49c-d378-4dba-a5d1-1f1d0be5df03","Type":"ContainerStarted","Data":"51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72"}
Oct 31 04:28:17.594735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.594684    5831 generic.go:356] "Generic (PLEG): container finished" podID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerID="ba2ba04d0db412d5745b9fd4437ba48a6b95d0116cb6dd1b28f6138d0acdf4e6" exitCode=0
Oct 31 04:28:17.594872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.594778    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" event={"ID":"b22186f4-aee3-49d9-b2b7-753e7d4be148","Type":"ContainerDied","Data":"ba2ba04d0db412d5745b9fd4437ba48a6b95d0116cb6dd1b28f6138d0acdf4e6"}
Oct 31 04:28:17.699456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.699391    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9" podStartSLOduration=1.699371499 podStartE2EDuration="1.699371499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:17.697496673 +0000 UTC m=+4847.712058392" watchObservedRunningTime="2026-10-31 04:28:17.699371499 +0000 UTC m=+4847.713933208"
Oct 31 04:28:17.699972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.699948    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 04:28:17.782563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.780707    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\" (UniqueName: \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\") pod \"b22186f4-aee3-49d9-b2b7-753e7d4be148\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") "
Oct 31 04:28:17.782563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.780818    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\") pod \"b22186f4-aee3-49d9-b2b7-753e7d4be148\" (UID: \"b22186f4-aee3-49d9-b2b7-753e7d4be148\") "
Oct 31 04:28:17.782563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.781566    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume" pod "b22186f4-aee3-49d9-b2b7-753e7d4be148" (UID: "b22186f4-aee3-49d9-b2b7-753e7d4be148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:17.784224 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.784197    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c" pod "b22186f4-aee3-49d9-b2b7-753e7d4be148" (UID: "b22186f4-aee3-49d9-b2b7-753e7d4be148"). InnerVolumeSpecName "kube-api-access-z2f7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:17.816760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.816728    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 04:28:17.882236 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882201    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\") pod \"57de5143-5837-4403-b4f0-63d7fbd4482b\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") "
Oct 31 04:28:17.882498 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882359    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\" (UniqueName: \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\") pod \"57de5143-5837-4403-b4f0-63d7fbd4482b\" (UID: \"57de5143-5837-4403-b4f0-63d7fbd4482b\") "
Oct 31 04:28:17.882628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882606    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume" pod "57de5143-5837-4403-b4f0-63d7fbd4482b" (UID: "57de5143-5837-4403-b4f0-63d7fbd4482b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:17.882702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882686    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b22186f4-aee3-49d9-b2b7-753e7d4be148-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:17.882759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882705    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2f7c\" (UniqueName: \"kubernetes.io/projected/b22186f4-aee3-49d9-b2b7-753e7d4be148-kube-api-access-z2f7c\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:17.882759 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.882715    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57de5143-5837-4403-b4f0-63d7fbd4482b-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:17.884622 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.884602    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv" pod "57de5143-5837-4403-b4f0-63d7fbd4482b" (UID: "57de5143-5837-4403-b4f0-63d7fbd4482b"). InnerVolumeSpecName "kube-api-access-m56sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:17.984021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:17.983981    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56sv\" (UniqueName: \"kubernetes.io/projected/57de5143-5837-4403-b4f0-63d7fbd4482b-kube-api-access-m56sv\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:18.469772 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.469724    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.33:50052\" within 1s: context deadline exceeded"
Oct 31 04:28:18.470193 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:18.469792    5831 logging.go:55] [core] [Channel #18480 SubChannel #18482]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.33:50052", ServerName: "10.132.1.33:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.33:50052: operation was canceled"
Oct 31 04:28:18.470193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.469824    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 04:28:18.603937 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.603903    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"
Oct 31 04:28:18.604143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.603932    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k" event={"ID":"b22186f4-aee3-49d9-b2b7-753e7d4be148","Type":"ContainerDied","Data":"6b28a1bccb4fab4ef04e2fe1911f4265b256f02cf753a4d27abaac416745fb3d"}
Oct 31 04:28:18.604143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.604011    5831 scope.go:122] "RemoveContainer" containerID="ba2ba04d0db412d5745b9fd4437ba48a6b95d0116cb6dd1b28f6138d0acdf4e6"
Oct 31 04:28:18.605962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.605930    5831 generic.go:356] "Generic (PLEG): container finished" podID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerID="d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4" exitCode=0
Oct 31 04:28:18.606121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.605986    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"
Oct 31 04:28:18.606121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.606007    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" event={"ID":"57de5143-5837-4403-b4f0-63d7fbd4482b","Type":"ContainerDied","Data":"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"}
Oct 31 04:28:18.606121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.606108    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd" event={"ID":"57de5143-5837-4403-b4f0-63d7fbd4482b","Type":"ContainerDied","Data":"dc8f7bb7ba3a899f3df9372cb17271bf7c5887db9d46f2f582e9bf13926dd23e"}
Oct 31 04:28:18.608436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.608408    5831 generic.go:356] "Generic (PLEG): container finished" podID="de5fe49c-d378-4dba-a5d1-1f1d0be5df03" containerID="e2d5c7b051e536efc82895ca5d079e98749bd64d4d7032656232666b866b8aec" exitCode=0
Oct 31 04:28:18.608572 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.608488    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9" event={"ID":"de5fe49c-d378-4dba-a5d1-1f1d0be5df03","Type":"ContainerDied","Data":"e2d5c7b051e536efc82895ca5d079e98749bd64d4d7032656232666b866b8aec"}
Oct 31 04:28:18.617615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.617596    5831 scope.go:122] "RemoveContainer" containerID="d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"
Oct 31 04:28:18.627901 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.627883    5831 scope.go:122] "RemoveContainer" containerID="d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"
Oct 31 04:28:18.628210 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:18.628186    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4\": container with ID starting with d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4 not found: ID does not exist" containerID="d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"
Oct 31 04:28:18.628254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.628219    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4"} err="failed to get container status \"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4\": rpc error: code = NotFound desc = could not find container \"d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4\": container with ID starting with d7fcb57679f702af9ea4cdad9e3951251d6f458e2a95d7b7ec8f5b7eaff08cb4 not found: ID does not exist"
Oct 31 04:28:18.755333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.755257    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 04:28:18.801008 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.800976    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-502ec2-7cbd64d88c-gbbsd"]
Oct 31 04:28:18.857504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.857465    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 04:28:18.871123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:18.861586    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-502ec2-5b958857d9-jcj9k"]
Oct 31 04:28:19.749999 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.749975    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:19.802054 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.802023    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\" (UniqueName: \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\") pod \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") "
Oct 31 04:28:19.802229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.802071    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\") pod \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") "
Oct 31 04:28:19.802229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.802143    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\") pod \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\" (UID: \"de5fe49c-d378-4dba-a5d1-1f1d0be5df03\") "
Oct 31 04:28:19.802503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.802481    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca" pod "de5fe49c-d378-4dba-a5d1-1f1d0be5df03" (UID: "de5fe49c-d378-4dba-a5d1-1f1d0be5df03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:19.804445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.804419    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx" pod "de5fe49c-d378-4dba-a5d1-1f1d0be5df03" (UID: "de5fe49c-d378-4dba-a5d1-1f1d0be5df03"). InnerVolumeSpecName "kube-api-access-4dcrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:19.804546 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.804451    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca" pod "de5fe49c-d378-4dba-a5d1-1f1d0be5df03" (UID: "de5fe49c-d378-4dba-a5d1-1f1d0be5df03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:19.903319 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.903290    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:19.903319 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.903319    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcrx\" (UniqueName: \"kubernetes.io/projected/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-kube-api-access-4dcrx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:19.903494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:19.903331    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5fe49c-d378-4dba-a5d1-1f1d0be5df03-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.120272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.120236    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 04:28:20.135820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.135748    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" path="/var/lib/kubelet/pods/57de5143-5837-4403-b4f0-63d7fbd4482b/volumes"
Oct 31 04:28:20.136147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.136133    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" path="/var/lib/kubelet/pods/b22186f4-aee3-49d9-b2b7-753e7d4be148/volumes"
Oct 31 04:28:20.206093 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.206060    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\" (UniqueName: \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\") pod \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") "
Oct 31 04:28:20.206278 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.206111    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\") pod \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\" (UID: \"0f908d42-2b5c-43ec-8c4a-8bb2301424f9\") "
Oct 31 04:28:20.206571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.206544    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume" pod "0f908d42-2b5c-43ec-8c4a-8bb2301424f9" (UID: "0f908d42-2b5c-43ec-8c4a-8bb2301424f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:20.208450 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.208431    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff" pod "0f908d42-2b5c-43ec-8c4a-8bb2301424f9" (UID: "0f908d42-2b5c-43ec-8c4a-8bb2301424f9"). InnerVolumeSpecName "kube-api-access-l6lff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:20.232646 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.232612    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 04:28:20.307156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.307131    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\" (UniqueName: \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\") pod \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") "
Oct 31 04:28:20.307281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.307266    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\") pod \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\" (UID: \"3e6a16f7-496b-40aa-ac25-1d9588d2edb7\") "
Oct 31 04:28:20.307452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.307439    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6lff\" (UniqueName: \"kubernetes.io/projected/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-kube-api-access-l6lff\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.307504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.307455    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f908d42-2b5c-43ec-8c4a-8bb2301424f9-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.307638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.307616    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume" pod "3e6a16f7-496b-40aa-ac25-1d9588d2edb7" (UID: "3e6a16f7-496b-40aa-ac25-1d9588d2edb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:20.309336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.309320    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48" pod "3e6a16f7-496b-40aa-ac25-1d9588d2edb7" (UID: "3e6a16f7-496b-40aa-ac25-1d9588d2edb7"). InnerVolumeSpecName "kube-api-access-2zf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:20.409213 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.409178    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zf48\" (UniqueName: \"kubernetes.io/projected/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-kube-api-access-2zf48\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.409347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.409220    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6a16f7-496b-40aa-ac25-1d9588d2edb7-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.549240 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.549214    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 04:28:20.610597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610560    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.610802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610638    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.610802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610686    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.610802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610711    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.610802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610768    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.611041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610855    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.611041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610888    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.611041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610961    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.611041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.610994    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\") pod \"d389e7f6-79c7-4631-a3d0-51f7989592dd\" (UID: \"d389e7f6-79c7-4631-a3d0-51f7989592dd\") "
Oct 31 04:28:20.611311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611011    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:20.611311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611084    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:20.611422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611330    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.611422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611332    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:20.611422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611351    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.611422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611395    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:20.611803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.611773    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:20.613230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.613207    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:20.613935 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.613904    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:28:20.614016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.613992    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:20.614016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.614007    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2" pod "d389e7f6-79c7-4631-a3d0-51f7989592dd" (UID: "d389e7f6-79c7-4631-a3d0-51f7989592dd"). InnerVolumeSpecName "kube-api-access-5plq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:20.625137 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.625110    5831 generic.go:356] "Generic (PLEG): container finished" podID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerID="c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101" exitCode=0
Oct 31 04:28:20.625255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.625187    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"
Oct 31 04:28:20.625255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.625194    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerDied","Data":"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"}
Oct 31 04:28:20.625379 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.625265    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb" event={"ID":"d389e7f6-79c7-4631-a3d0-51f7989592dd","Type":"ContainerDied","Data":"1963dc59e6f0d79ce35002cc1dd49a85ba2bb786b0d865b9d05b92570ea2a4d0"}
Oct 31 04:28:20.625379 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.625298    5831 scope.go:122] "RemoveContainer" containerID="c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"
Oct 31 04:28:20.626795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.626742    5831 generic.go:356] "Generic (PLEG): container finished" podID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerID="f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0" exitCode=0
Oct 31 04:28:20.626880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.626795    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"
Oct 31 04:28:20.626880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.626818    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" event={"ID":"3e6a16f7-496b-40aa-ac25-1d9588d2edb7","Type":"ContainerDied","Data":"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"}
Oct 31 04:28:20.626880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.626863    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr" event={"ID":"3e6a16f7-496b-40aa-ac25-1d9588d2edb7","Type":"ContainerDied","Data":"db3f830c9e9485edcddf7563a3ecfc12daa3582fc526fb44b658fd9d935d83da"}
Oct 31 04:28:20.628493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.628471    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9" event={"ID":"de5fe49c-d378-4dba-a5d1-1f1d0be5df03","Type":"ContainerDied","Data":"51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72"}
Oct 31 04:28:20.628706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.628514    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e1b67ac7e25d6d6a612e5de7e86dfa2f021894f697e37a05275e57e8bf2d72"
Oct 31 04:28:20.628706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.628599    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"
Oct 31 04:28:20.630855 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.630829    5831 generic.go:356] "Generic (PLEG): container finished" podID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerID="4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab" exitCode=0
Oct 31 04:28:20.630977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.630870    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" event={"ID":"0f908d42-2b5c-43ec-8c4a-8bb2301424f9","Type":"ContainerDied","Data":"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"}
Oct 31 04:28:20.630977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.630906    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr" event={"ID":"0f908d42-2b5c-43ec-8c4a-8bb2301424f9","Type":"ContainerDied","Data":"320813e119dd98a6ba9e569f7052bf64079e2b6ff7805edca786c5fd4a3925fd"}
Oct 31 04:28:20.630977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.630967    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"
Oct 31 04:28:20.637028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.637007    5831 scope.go:122] "RemoveContainer" containerID="77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"
Oct 31 04:28:20.646743 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.646686    5831 scope.go:122] "RemoveContainer" containerID="c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"
Oct 31 04:28:20.646978 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:20.646960    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101\": container with ID starting with c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101 not found: ID does not exist" containerID="c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"
Oct 31 04:28:20.647025 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.646987    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101"} err="failed to get container status \"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101\": rpc error: code = NotFound desc = could not find container \"c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101\": container with ID starting with c457b5f02504d9b3669dd6aa9b0be656d45f535505f3da7ae74a997121b82101 not found: ID does not exist"
Oct 31 04:28:20.647025 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.647005    5831 scope.go:122] "RemoveContainer" containerID="77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"
Oct 31 04:28:20.647222 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:20.647204    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de\": container with ID starting with 77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de not found: ID does not exist" containerID="77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"
Oct 31 04:28:20.647265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.647228    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de"} err="failed to get container status \"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de\": rpc error: code = NotFound desc = could not find container \"77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de\": container with ID starting with 77eb0a948ea7236d4d1a08cf826ff133ca4cf23a13aaccfeedb1681ea9ae19de not found: ID does not exist"
Oct 31 04:28:20.647265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.647245    5831 scope.go:122] "RemoveContainer" containerID="f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"
Oct 31 04:28:20.655823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.655802    5831 scope.go:122] "RemoveContainer" containerID="f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"
Oct 31 04:28:20.656074 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:20.656053    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0\": container with ID starting with f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0 not found: ID does not exist" containerID="f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"
Oct 31 04:28:20.656164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.656081    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0"} err="failed to get container status \"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0\": rpc error: code = NotFound desc = could not find container \"f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0\": container with ID starting with f6b8f7166aa11f2ca25dc9a3349caf04bf410b3d3253d98bff2f3c629837a9a0 not found: ID does not exist"
Oct 31 04:28:20.656164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.656097    5831 scope.go:122] "RemoveContainer" containerID="4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"
Oct 31 04:28:20.667229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.667205    5831 scope.go:122] "RemoveContainer" containerID="4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"
Oct 31 04:28:20.667583 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:20.667560    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab\": container with ID starting with 4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab not found: ID does not exist" containerID="4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"
Oct 31 04:28:20.667663 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.667591    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab"} err="failed to get container status \"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab\": rpc error: code = NotFound desc = could not find container \"4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab\": container with ID starting with 4ad5b91f841cb1044f626341bdf799c1e7ee541390aa7d8ffb14bfa99c13caab not found: ID does not exist"
Oct 31 04:28:20.712872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712840    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.712872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712866    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.712872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712877    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.713186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712886    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d389e7f6-79c7-4631-a3d0-51f7989592dd-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.713186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712895    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.713186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712904    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d389e7f6-79c7-4631-a3d0-51f7989592dd-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.713186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.712912    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plq2\" (UniqueName: \"kubernetes.io/projected/d389e7f6-79c7-4631-a3d0-51f7989592dd-kube-api-access-5plq2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:20.764994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.764953    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 04:28:20.768214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.768186    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c0d50c-8fdc57f-cpzwr"]
Oct 31 04:28:20.920895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.920789    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 04:28:20.923397 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:20.923368    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c0d50c-74c58cbdd5-f4tnr"]
Oct 31 04:28:21.022119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:21.022085    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 04:28:21.027743 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:21.027702    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-502ec2-openshift-default-68c9756987-2m7tb"]
Oct 31 04:28:22.135616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:22.135577    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" path="/var/lib/kubelet/pods/0f908d42-2b5c-43ec-8c4a-8bb2301424f9/volumes"
Oct 31 04:28:22.136108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:22.136080    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" path="/var/lib/kubelet/pods/3e6a16f7-496b-40aa-ac25-1d9588d2edb7/volumes"
Oct 31 04:28:22.136689 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:22.136671    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" path="/var/lib/kubelet/pods/d389e7f6-79c7-4631-a3d0-51f7989592dd/volumes"
Oct 31 04:28:22.317711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:22.317674    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:28:23.764913 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:23.764881    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 04:28:23.765315 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:23.765094    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" containerID="cri-o://274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c" gracePeriod=30
Oct 31 04:28:24.862297 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862252    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"]
Oct 31 04:28:24.862934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862913    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862938    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862953    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862960    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862982    5831 cpu_manager.go:414] "containerMap: removing container" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.862991    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863022    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing"
Oct 31 04:28:24.863032 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863030    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863043    5831 cpu_manager.go:414] "containerMap: removing container" podUID="de5fe49c-d378-4dba-a5d1-1f1d0be5df03" containerName="revoke-keys"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863052    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5fe49c-d378-4dba-a5d1-1f1d0be5df03" containerName="revoke-keys"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863065    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863073    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863195    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="57de5143-5837-4403-b4f0-63d7fbd4482b" containerName="payload-pre-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863213    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6a16f7-496b-40aa-ac25-1d9588d2edb7" containerName="payload-pre-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863224    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5fe49c-d378-4dba-a5d1-1f1d0be5df03" containerName="revoke-keys"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863234    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22186f4-aee3-49d9-b2b7-753e7d4be148" containerName="payload-processing"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863246    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:24.863443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.863256    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f908d42-2b5c-43ec-8c4a-8bb2301424f9" containerName="payload-processing"
Oct 31 04:28:24.868379 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.868358    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:24.918524 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.918495    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"]
Oct 31 04:28:24.956760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.956729    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:24.956936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.956784    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:24.956936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.956882    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjkk\" (UniqueName: \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:24.961520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.961495    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:28:24.961687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.961596    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:28:24.961687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.961645    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:28:24.962832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:24.962816    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:28:25.057486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.057455    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.057684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.057559    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.057684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.057608    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjkk\" (UniqueName: \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.058165 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.058139    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.059905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.059881    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.110335 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.110308    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjkk\" (UniqueName: \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\") pod \"maas-api-revoke-sub-515c7eca-qdcps\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.181630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.181603    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:25.348396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.348357    5831 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:28:25.348598 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.348431    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:28:25.549297 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.549274    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"]
Oct 31 04:28:25.552475 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:25.552446    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc49af2_1de3_45b6_a5ea_de0f7ab98b08.slice/crio-8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d WatchSource:0}: Error finding container 8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d: Status 404 returned error can't find the container with id 8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d
Oct 31 04:28:25.671234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:25.671198    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps" event={"ID":"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08","Type":"ContainerStarted","Data":"8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d"}
Oct 31 04:28:26.679397 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:26.679359    5831 generic.go:356] "Generic (PLEG): container finished" podID="0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" containerID="a92e69ddb3abffb9b4ecb79d35b9b72e68b603582fd0b56ee9cff5430e12bdc6" exitCode=0
Oct 31 04:28:26.679397 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:26.679400    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps" event={"ID":"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08","Type":"ContainerDied","Data":"a92e69ddb3abffb9b4ecb79d35b9b72e68b603582fd0b56ee9cff5430e12bdc6"}
Oct 31 04:28:27.841341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.841311    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:27.886607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.886575    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\") pod \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") "
Oct 31 04:28:27.886807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.886663    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\" (UniqueName: \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\") pod \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") "
Oct 31 04:28:27.886807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.886757    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\") pod \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\" (UID: \"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08\") "
Oct 31 04:28:27.887221 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.887183    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca" pod "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" (UID: "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:27.889210 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.889182    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca" pod "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" (UID: "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:27.890298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.889743    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk" pod "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" (UID: "0dc49af2-1de3-45b6-a5ea-de0f7ab98b08"). InnerVolumeSpecName "kube-api-access-qbjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:27.987933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.987860    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:27.987933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.987891    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:27.987933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:27.987900    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjkk\" (UniqueName: \"kubernetes.io/projected/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08-kube-api-access-qbjkk\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:28.344665 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.344556    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"]
Oct 31 04:28:28.345057 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345043    5831 cpu_manager.go:414] "containerMap: removing container" podUID="0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" containerName="revoke-keys"
Oct 31 04:28:28.345103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345061    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" containerName="revoke-keys"
Oct 31 04:28:28.345103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345078    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:28.345103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345082    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:28.345205 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345147    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d389e7f6-79c7-4631-a3d0-51f7989592dd" containerName="istio-proxy"
Oct 31 04:28:28.345205 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.345160    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" containerName="revoke-keys"
Oct 31 04:28:28.348795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.348775    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.392259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.392222    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.392450 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.392383    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.392549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.392505    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4sb\" (UniqueName: \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.401910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.401874    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"]
Oct 31 04:28:28.493953 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.493913    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.494147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.494018    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.494147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.494093    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4sb\" (UniqueName: \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.495075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.495043    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.496644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.496621    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.546878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.546851    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4sb\" (UniqueName: \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\") pod \"maas-api-revoke-sub-aa2eb7e3-8g6xl\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.663481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.663392    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:28.703617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.703572    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps" event={"ID":"0dc49af2-1de3-45b6-a5ea-de0f7ab98b08","Type":"ContainerDied","Data":"8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d"}
Oct 31 04:28:28.703937 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.703918    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f10b2abf5c49149073db405e1b4728d64e67801f9eb38c14dbb0537f9efd47d"
Oct 31 04:28:28.704148 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:28.704117    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"
Oct 31 04:28:29.037412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.037387    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 04:28:29.078036 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.078010    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"]
Oct 31 04:28:29.080766 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:29.080732    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9687d6_a017_4c81_a444_e30cffd75bb0.slice/crio-e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a WatchSource:0}: Error finding container e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a: Status 404 returned error can't find the container with id e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a
Oct 31 04:28:29.100044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100012    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100182 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100076    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100182 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100134    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100194    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100228    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100255    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100316    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100384    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100428    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\") pod \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\" (UID: \"7280b668-d83a-4977-9cdc-1a394ac5b9f7\") "
Oct 31 04:28:29.100723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100483    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:29.100842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100751    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.100983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.100962    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:29.101222 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.101007    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:29.101294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.101275    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:29.101581 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.101556    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:29.103780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.103758    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:29.103866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.103833    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "kube-api-access-tcl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:29.104943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.104920    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:28:29.105733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.105711    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token" pod "7280b668-d83a-4977-9cdc-1a394ac5b9f7" (UID: "7280b668-d83a-4977-9cdc-1a394ac5b9f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202295    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202324    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcl6m\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-kube-api-access-tcl6m\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202334    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202342    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202351    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202360    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202370    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280b668-d83a-4977-9cdc-1a394ac5b9f7-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.202384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.202378    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7280b668-d83a-4977-9cdc-1a394ac5b9f7-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:29.218754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.218714    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 04:28:29.317251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.317214    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 04:28:29.422809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.422765    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 04:28:29.713111 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.713074    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl" event={"ID":"2c9687d6-a017-4c81-a444-e30cffd75bb0","Type":"ContainerStarted","Data":"42485c7dd6acd0bd5c0c48c49e902f1bc701983112fdd1df8705314fb340ecd6"}
Oct 31 04:28:29.713330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.713127    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl" event={"ID":"2c9687d6-a017-4c81-a444-e30cffd75bb0","Type":"ContainerStarted","Data":"e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a"}
Oct 31 04:28:29.714907 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.714881    5831 generic.go:356] "Generic (PLEG): container finished" podID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerID="274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c" exitCode=0
Oct 31 04:28:29.715072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.714934    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"
Oct 31 04:28:29.715072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.714963    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerDied","Data":"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"}
Oct 31 04:28:29.715072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.715011    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc" event={"ID":"7280b668-d83a-4977-9cdc-1a394ac5b9f7","Type":"ContainerDied","Data":"0179578451fcefe4b282e13ba7e61c6ef51bda36af990ec2c8d0aa61b82905a3"}
Oct 31 04:28:29.715072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.715033    5831 scope.go:122] "RemoveContainer" containerID="274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"
Oct 31 04:28:29.725492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.725476    5831 scope.go:122] "RemoveContainer" containerID="3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"
Oct 31 04:28:29.734687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.734668    5831 scope.go:122] "RemoveContainer" containerID="274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"
Oct 31 04:28:29.734982 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:29.734959    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c\": container with ID starting with 274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c not found: ID does not exist" containerID="274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"
Oct 31 04:28:29.735075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.734995    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c"} err="failed to get container status \"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c\": rpc error: code = NotFound desc = could not find container \"274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c\": container with ID starting with 274b92ef8b961b5e6099aa59d8fb1cd1e4bdbb97e7b327e46f2e27030687a23c not found: ID does not exist"
Oct 31 04:28:29.735075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.735014    5831 scope.go:122] "RemoveContainer" containerID="3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"
Oct 31 04:28:29.735225 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:29.735210    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64\": container with ID starting with 3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64 not found: ID does not exist" containerID="3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"
Oct 31 04:28:29.735292 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.735228    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64"} err="failed to get container status \"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64\": rpc error: code = NotFound desc = could not find container \"3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64\": container with ID starting with 3a29efb3abe2b21a010f6543f66972a40e594f7273d5567e374f2c298c44ba64 not found: ID does not exist"
Oct 31 04:28:29.819819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.819769    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl" podStartSLOduration=1.8197565770000002 podStartE2EDuration="1.819756577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:29.818129633 +0000 UTC m=+4859.832691342" watchObservedRunningTime="2026-10-31 04:28:29.819756577 +0000 UTC m=+4859.834318284"
Oct 31 04:28:29.922178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.922147    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 04:28:29.926396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:29.926369    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c0d50c-openshift-default-59d947f867-ngmwc"]
Oct 31 04:28:30.135767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:30.135737    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" path="/var/lib/kubelet/pods/7280b668-d83a-4977-9cdc-1a394ac5b9f7/volumes"
Oct 31 04:28:30.724736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:30.724698    5831 generic.go:356] "Generic (PLEG): container finished" podID="2c9687d6-a017-4c81-a444-e30cffd75bb0" containerID="42485c7dd6acd0bd5c0c48c49e902f1bc701983112fdd1df8705314fb340ecd6" exitCode=0
Oct 31 04:28:30.724914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:30.724781    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl" event={"ID":"2c9687d6-a017-4c81-a444-e30cffd75bb0","Type":"ContainerDied","Data":"42485c7dd6acd0bd5c0c48c49e902f1bc701983112fdd1df8705314fb340ecd6"}
Oct 31 04:28:31.875818 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.875795    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:31.935220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.935188    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\") pod \"2c9687d6-a017-4c81-a444-e30cffd75bb0\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") "
Oct 31 04:28:31.935380 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.935264    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\" (UniqueName: \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\") pod \"2c9687d6-a017-4c81-a444-e30cffd75bb0\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") "
Oct 31 04:28:31.935380 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.935299    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\") pod \"2c9687d6-a017-4c81-a444-e30cffd75bb0\" (UID: \"2c9687d6-a017-4c81-a444-e30cffd75bb0\") "
Oct 31 04:28:31.935742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.935719    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca" pod "2c9687d6-a017-4c81-a444-e30cffd75bb0" (UID: "2c9687d6-a017-4c81-a444-e30cffd75bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:31.937829 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.937798    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb" pod "2c9687d6-a017-4c81-a444-e30cffd75bb0" (UID: "2c9687d6-a017-4c81-a444-e30cffd75bb0"). InnerVolumeSpecName "kube-api-access-qv4sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:31.937949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:31.937933    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca" pod "2c9687d6-a017-4c81-a444-e30cffd75bb0" (UID: "2c9687d6-a017-4c81-a444-e30cffd75bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:32.036208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.036120    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:32.036208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.036150    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4sb\" (UniqueName: \"kubernetes.io/projected/2c9687d6-a017-4c81-a444-e30cffd75bb0-kube-api-access-qv4sb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:32.036208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.036162    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9687d6-a017-4c81-a444-e30cffd75bb0-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:32.739197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.739171    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"
Oct 31 04:28:32.739452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.739207    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl" event={"ID":"2c9687d6-a017-4c81-a444-e30cffd75bb0","Type":"ContainerDied","Data":"e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a"}
Oct 31 04:28:32.739452 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:32.739252    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b870b9543e7c7695d2b337be0af0d0e874287458a5de0ad8c8043f411bf88a"
Oct 31 04:28:34.318321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:34.318277    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing" containerID="cri-o://703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895" gracePeriod=25
Oct 31 04:28:34.423058 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:34.423017    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing" containerID="cri-o://574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e" gracePeriod=25
Oct 31 04:28:36.397170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.397127    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 04:28:36.504435 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.504386    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 04:28:36.745952 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.745917    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"]
Oct 31 04:28:36.746623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746595    5831 cpu_manager.go:414] "containerMap: removing container" podUID="2c9687d6-a017-4c81-a444-e30cffd75bb0" containerName="revoke-keys"
Oct 31 04:28:36.746623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746616    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9687d6-a017-4c81-a444-e30cffd75bb0" containerName="revoke-keys"
Oct 31 04:28:36.746623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746629    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746634    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746646    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746651    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746727    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9687d6-a017-4c81-a444-e30cffd75bb0" containerName="revoke-keys"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746735    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.746823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.746743    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7280b668-d83a-4977-9cdc-1a394ac5b9f7" containerName="istio-proxy"
Oct 31 04:28:36.749925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.749902    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.800888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.800349    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:28:36.800888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.800404    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:28:36.800888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.800446    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:28:36.800888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.800514    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:28:36.803893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.803868    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"]
Oct 31 04:28:36.885228 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.885198    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.885427 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.885239    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ztp\" (UniqueName: \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.885507 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.885488    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.986712 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.986671    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.986921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.986730    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.986921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.986772    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ztp\" (UniqueName: \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.987465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.987440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:36.989855 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:36.989830    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:37.039586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.039490    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ztp\" (UniqueName: \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\") pod \"maas-api-revoke-sub-c8a7e02b-6g58h\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:37.063879 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.063851    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:37.198292 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.198254    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:28:37.451600 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.451567    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"]
Oct 31 04:28:37.453949 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:28:37.453917    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56176d7_3cb7_4685_89cb_1192a32124a0.slice/crio-dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507 WatchSource:0}: Error finding container dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507: Status 404 returned error can't find the container with id dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507
Oct 31 04:28:37.633793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.633678    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:37.774544 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:37.774489    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h" event={"ID":"d56176d7-3cb7-4685-89cb-1192a32124a0","Type":"ContainerStarted","Data":"dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507"}
Oct 31 04:28:38.648083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:38.648040    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:38.782674 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:38.782583    5831 generic.go:356] "Generic (PLEG): container finished" podID="d56176d7-3cb7-4685-89cb-1192a32124a0" containerID="8aa837b57ff67d18a6b3895a5bce9999f034ba97bd8567b0b45fd2c1115cce14" exitCode=0
Oct 31 04:28:38.782861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:38.782677    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h" event={"ID":"d56176d7-3cb7-4685-89cb-1192a32124a0","Type":"ContainerDied","Data":"8aa837b57ff67d18a6b3895a5bce9999f034ba97bd8567b0b45fd2c1115cce14"}
Oct 31 04:28:39.937791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:39.937751    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:40.029248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.029216    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\" (UniqueName: \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\") pod \"d56176d7-3cb7-4685-89cb-1192a32124a0\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") "
Oct 31 04:28:40.029248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.029254    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\") pod \"d56176d7-3cb7-4685-89cb-1192a32124a0\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") "
Oct 31 04:28:40.029454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.029341    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\") pod \"d56176d7-3cb7-4685-89cb-1192a32124a0\" (UID: \"d56176d7-3cb7-4685-89cb-1192a32124a0\") "
Oct 31 04:28:40.029847 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.029818    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca" pod "d56176d7-3cb7-4685-89cb-1192a32124a0" (UID: "d56176d7-3cb7-4685-89cb-1192a32124a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:40.031713 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.031686    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp" pod "d56176d7-3cb7-4685-89cb-1192a32124a0" (UID: "d56176d7-3cb7-4685-89cb-1192a32124a0"). InnerVolumeSpecName "kube-api-access-p5ztp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:40.031824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.031719    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca" pod "d56176d7-3cb7-4685-89cb-1192a32124a0" (UID: "d56176d7-3cb7-4685-89cb-1192a32124a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:40.130092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.130011    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:40.130092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.130037    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ztp\" (UniqueName: \"kubernetes.io/projected/d56176d7-3cb7-4685-89cb-1192a32124a0-kube-api-access-p5ztp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:40.130092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.130049    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56176d7-3cb7-4685-89cb-1192a32124a0-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:40.797005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.796960    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h" event={"ID":"d56176d7-3cb7-4685-89cb-1192a32124a0","Type":"ContainerDied","Data":"dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507"}
Oct 31 04:28:40.799619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.797035    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbb4a6f7af883360aa11831004c3a3311ead50481504f54b2a055e0765179507"
Oct 31 04:28:40.799619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:40.796976    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"
Oct 31 04:28:41.398004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:41.397965    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing" containerID="cri-o://b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c" gracePeriod=25
Oct 31 04:28:41.505257 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:41.505200    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing" containerID="cri-o://c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242" gracePeriod=25
Oct 31 04:28:42.978180 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:42.978143    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:28:44.240978 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.240916    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main" containerID="cri-o://8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee" gracePeriod=45
Oct 31 04:28:44.506740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.506674    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 04:28:44.584863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.584832    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.584863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.584870    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.585138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.584890    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.585138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.584948    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\" (UniqueName: \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.585138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585037    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.585138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585068    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\") pod \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\" (UID: \"f4f15a43-2b9e-4b8e-88b2-33982e111aca\") "
Oct 31 04:28:44.585337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585181    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.585337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585238    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.585454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585433    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.585490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585456    5831 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-home\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.585490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.585477    5831 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-model-cache\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.587484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.587456    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:44.587484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.587458    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "kube-api-access-tcl5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:44.587678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.587514    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm" pod "f4f15a43-2b9e-4b8e-88b2-33982e111aca" (UID: "f4f15a43-2b9e-4b8e-88b2-33982e111aca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.686451 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.686420    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.686451 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.686450    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcl5r\" (UniqueName: \"kubernetes.io/projected/f4f15a43-2b9e-4b8e-88b2-33982e111aca-kube-api-access-tcl5r\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.686694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.686460    5831 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-dshm\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.686694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.686471    5831 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f15a43-2b9e-4b8e-88b2-33982e111aca-tmp-dir\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.821698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.821605    5831 generic.go:356] "Generic (PLEG): container finished" podID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerID="8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee" exitCode=0
Oct 31 04:28:44.821698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.821654    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"
Oct 31 04:28:44.821941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.821651    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" event={"ID":"f4f15a43-2b9e-4b8e-88b2-33982e111aca","Type":"ContainerDied","Data":"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"}
Oct 31 04:28:44.821941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.821773    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726" event={"ID":"f4f15a43-2b9e-4b8e-88b2-33982e111aca","Type":"ContainerDied","Data":"1f7fb5312e575066dee54d36a6b60553e64613bb698cadd3c3885ce5ad1759b7"}
Oct 31 04:28:44.821941 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.821794    5831 scope.go:122] "RemoveContainer" containerID="8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"
Oct 31 04:28:44.832613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.832593    5831 scope.go:122] "RemoveContainer" containerID="8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"
Oct 31 04:28:44.832917 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:44.832896    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee\": container with ID starting with 8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee not found: ID does not exist" containerID="8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"
Oct 31 04:28:44.832997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.832926    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee"} err="failed to get container status \"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee\": rpc error: code = NotFound desc = could not find container \"8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee\": container with ID starting with 8572ebf6efbbaa961301f6c1881ac1b5fe4d527fe1a1d8a69e6616dea804dfee not found: ID does not exist"
Oct 31 04:28:44.974152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.974119    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 04:28:44.976389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:44.976365    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/premium-w4-9360eb-kserve-5cd6f4bdc7-bs726"]
Oct 31 04:28:46.140520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:46.140484    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" path="/var/lib/kubelet/pods/f4f15a43-2b9e-4b8e-88b2-33982e111aca/volumes"
Oct 31 04:28:47.633566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:47.633506    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:48.648885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:48.648846    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:49.740154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:49.740109    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:50.751083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:50.751041    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:54.587705 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.587629    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 04:28:54.686296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.686263    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 04:28:54.687284 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.687264    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\" (UniqueName: \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\") pod \"1d35f37a-64cf-492c-82d2-935cf283aa56\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") "
Oct 31 04:28:54.687406 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.687353    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\") pod \"1d35f37a-64cf-492c-82d2-935cf283aa56\" (UID: \"1d35f37a-64cf-492c-82d2-935cf283aa56\") "
Oct 31 04:28:54.687707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.687686    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume" pod "1d35f37a-64cf-492c-82d2-935cf283aa56" (UID: "1d35f37a-64cf-492c-82d2-935cf283aa56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:54.689596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.689564    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78" pod "1d35f37a-64cf-492c-82d2-935cf283aa56" (UID: "1d35f37a-64cf-492c-82d2-935cf283aa56"). InnerVolumeSpecName "kube-api-access-xbg78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:54.788072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.788033    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\" (UniqueName: \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\") pod \"f4c78470-f788-48aa-9b33-f6f23a76825a\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") "
Oct 31 04:28:54.788276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.788088    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\") pod \"f4c78470-f788-48aa-9b33-f6f23a76825a\" (UID: \"f4c78470-f788-48aa-9b33-f6f23a76825a\") "
Oct 31 04:28:54.788403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.788386    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d35f37a-64cf-492c-82d2-935cf283aa56-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:54.788472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.788404    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbg78\" (UniqueName: \"kubernetes.io/projected/1d35f37a-64cf-492c-82d2-935cf283aa56-kube-api-access-xbg78\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:54.788557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.788512    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume" pod "f4c78470-f788-48aa-9b33-f6f23a76825a" (UID: "f4c78470-f788-48aa-9b33-f6f23a76825a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:54.790337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.790311    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f" pod "f4c78470-f788-48aa-9b33-f6f23a76825a" (UID: "f4c78470-f788-48aa-9b33-f6f23a76825a"). InnerVolumeSpecName "kube-api-access-xq45f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:54.886583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.886463    5831 generic.go:356] "Generic (PLEG): container finished" podID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerID="703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895" exitCode=0
Oct 31 04:28:54.886583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.886509    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"
Oct 31 04:28:54.886583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.886545    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" event={"ID":"1d35f37a-64cf-492c-82d2-935cf283aa56","Type":"ContainerDied","Data":"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"}
Oct 31 04:28:54.886886 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.886607    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs" event={"ID":"1d35f37a-64cf-492c-82d2-935cf283aa56","Type":"ContainerDied","Data":"58bf34f22c5dfd0de5578b1ecdd7e02b92c3e5a75c90192f03526754b269a895"}
Oct 31 04:28:54.886886 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.886638    5831 scope.go:122] "RemoveContainer" containerID="703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"
Oct 31 04:28:54.888751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.888651    5831 generic.go:356] "Generic (PLEG): container finished" podID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerID="574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e" exitCode=0
Oct 31 04:28:54.888751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.888750    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" event={"ID":"f4c78470-f788-48aa-9b33-f6f23a76825a","Type":"ContainerDied","Data":"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"}
Oct 31 04:28:54.888751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.888792    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr" event={"ID":"f4c78470-f788-48aa-9b33-f6f23a76825a","Type":"ContainerDied","Data":"307cd102638e46bc3e5b35592c17dc4d889f41d1c55187500bba354c4abae44f"}
Oct 31 04:28:54.889068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.888853    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"
Oct 31 04:28:54.898621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.889611    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c78470-f788-48aa-9b33-f6f23a76825a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:54.898621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.889655    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq45f\" (UniqueName: \"kubernetes.io/projected/f4c78470-f788-48aa-9b33-f6f23a76825a-kube-api-access-xq45f\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:54.908339 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.908315    5831 scope.go:122] "RemoveContainer" containerID="703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"
Oct 31 04:28:54.908934 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:54.908904    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895\": container with ID starting with 703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895 not found: ID does not exist" containerID="703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"
Oct 31 04:28:54.909082 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.908947    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895"} err="failed to get container status \"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895\": rpc error: code = NotFound desc = could not find container \"703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895\": container with ID starting with 703d60b9e6583934a247f2f69965602faa38b31cb3c372ffee5e5019b5dfb895 not found: ID does not exist"
Oct 31 04:28:54.909082 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.908974    5831 scope.go:122] "RemoveContainer" containerID="574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"
Oct 31 04:28:54.920981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.920959    5831 scope.go:122] "RemoveContainer" containerID="574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"
Oct 31 04:28:54.921319 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:28:54.921299    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e\": container with ID starting with 574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e not found: ID does not exist" containerID="574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"
Oct 31 04:28:54.921388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:54.921326    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e"} err="failed to get container status \"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e\": rpc error: code = NotFound desc = could not find container \"574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e\": container with ID starting with 574d15d1cabf81548e7a9f001c20d2cb96b75296b30a3cea63df250e174e536e not found: ID does not exist"
Oct 31 04:28:55.038809 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:55.038773    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 04:28:55.044323 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:55.044298    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b59563-7b58f8bfc5-kl5rs"]
Oct 31 04:28:55.146552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:55.146444    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 04:28:55.150408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:55.150384    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b59563-6545bb797f-4szhr"]
Oct 31 04:28:56.135519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:56.135487    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" path="/var/lib/kubelet/pods/1d35f37a-64cf-492c-82d2-935cf283aa56/volumes"
Oct 31 04:28:56.135946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:56.135886    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" path="/var/lib/kubelet/pods/f4c78470-f788-48aa-9b33-f6f23a76825a/volumes"
Oct 31 04:28:59.497153 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497121    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497541    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d56176d7-3cb7-4685-89cb-1192a32124a0" containerName="revoke-keys"
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497554    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56176d7-3cb7-4685-89cb-1192a32124a0" containerName="revoke-keys"
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497569    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing"
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497574    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing"
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497582    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing"
Oct 31 04:28:59.497582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497587    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497604    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497609    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497674    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f15a43-2b9e-4b8e-88b2-33982e111aca" containerName="main"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497687    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56176d7-3cb7-4685-89cb-1192a32124a0" containerName="revoke-keys"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497694    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c78470-f788-48aa-9b33-f6f23a76825a" containerName="payload-pre-processing"
Oct 31 04:28:59.497800 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.497700    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d35f37a-64cf-492c-82d2-935cf283aa56" containerName="payload-processing"
Oct 31 04:28:59.502216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.502196    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.550458 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.550431    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 04:28:59.598937 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.598905    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:28:59.639123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639082    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639120    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639137    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639167    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcmz\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639303    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639350    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639405    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639424    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.639586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.639441    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.739982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.739945    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.739994    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740022    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740061    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcmz\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740102    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740188 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740148    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740196    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740225    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740252    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740403    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740769 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740560    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740769 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740651    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:28:59.740769 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740728    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.740905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.740801    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.741370 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.741345    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.742676 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.742658    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.742916 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.742900    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.793425 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.793350    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcmz\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.793801 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.793777    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:28:59.816511 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:28:59.816486    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:29:00.235790 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.235757    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:29:00.238633 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:29:00.238605    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fef98c1_865b_457f_baa2_990c4fd7532b.slice/crio-53447ab70f280ad3dd737c45e9dbc449890248718fc2e80ab89b75e5fc3d0eac WatchSource:0}: Error finding container 53447ab70f280ad3dd737c45e9dbc449890248718fc2e80ab89b75e5fc3d0eac: Status 404 returned error can't find the container with id 53447ab70f280ad3dd737c45e9dbc449890248718fc2e80ab89b75e5fc3d0eac
Oct 31 04:29:00.240785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.240752    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:00.240896 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.240821    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:00.240896 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.240849    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:00.752107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.752065    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:29:00.935876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.935837    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" event={"ID":"3fef98c1-865b-457f-baa2-990c4fd7532b","Type":"ContainerStarted","Data":"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"}
Oct 31 04:29:00.936059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:00.935894    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" event={"ID":"3fef98c1-865b-457f-baa2-990c4fd7532b","Type":"ContainerStarted","Data":"53447ab70f280ad3dd737c45e9dbc449890248718fc2e80ab89b75e5fc3d0eac"}
Oct 31 04:29:01.050053 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.049945    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" podStartSLOduration=2.049926958 podStartE2EDuration="2.049926958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:01.048448999 +0000 UTC m=+4891.063010721" watchObservedRunningTime="2026-10-31 04:29:01.049926958 +0000 UTC m=+4891.064488665"
Oct 31 04:29:01.705248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.704951    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 04:29:01.817506 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.817481    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:29:01.819275 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.819248    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 31 04:29:01.819419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.819293    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 31 04:29:01.840588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.840556    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 04:29:01.859700 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.859660    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\" (UniqueName: \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\") pod \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") "
Oct 31 04:29:01.859892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.859872    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\") pod \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\" (UID: \"cb3f1f8f-18ae-452a-9554-3bc17d7c5551\") "
Oct 31 04:29:01.860233 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.860208    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume" pod "cb3f1f8f-18ae-452a-9554-3bc17d7c5551" (UID: "cb3f1f8f-18ae-452a-9554-3bc17d7c5551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:01.860378 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.860350    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:01.862038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.862013    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf" pod "cb3f1f8f-18ae-452a-9554-3bc17d7c5551" (UID: "cb3f1f8f-18ae-452a-9554-3bc17d7c5551"). InnerVolumeSpecName "kube-api-access-ljhvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:01.946042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.945566    5831 generic.go:356] "Generic (PLEG): container finished" podID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerID="b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c" exitCode=0
Oct 31 04:29:01.946042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.945746    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"
Oct 31 04:29:01.946042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.945737    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" event={"ID":"cb3f1f8f-18ae-452a-9554-3bc17d7c5551","Type":"ContainerDied","Data":"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"}
Oct 31 04:29:01.946042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.945908    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh" event={"ID":"cb3f1f8f-18ae-452a-9554-3bc17d7c5551","Type":"ContainerDied","Data":"2585d25457f41346dc77cb9c64b13fd24476af5eb18563febef7905b82a4458c"}
Oct 31 04:29:01.946042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.945951    5831 scope.go:122] "RemoveContainer" containerID="b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"
Oct 31 04:29:01.948019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.947979    5831 generic.go:356] "Generic (PLEG): container finished" podID="5eb03491-33e0-4f12-b262-7042c4e58958" containerID="c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242" exitCode=0
Oct 31 04:29:01.948160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.948020    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"
Oct 31 04:29:01.948239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.948190    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" event={"ID":"5eb03491-33e0-4f12-b262-7042c4e58958","Type":"ContainerDied","Data":"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"}
Oct 31 04:29:01.948298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.948245    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx" event={"ID":"5eb03491-33e0-4f12-b262-7042c4e58958","Type":"ContainerDied","Data":"7aec27a841c3106e069b0a954d85cc456f6051fd156c89cce92bf2a286cde4cd"}
Oct 31 04:29:01.948641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.948620    5831 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" secret="" err="secret \"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\" not found"
Oct 31 04:29:01.957724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.957204    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:29:01.960163 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.960141    5831 scope.go:122] "RemoveContainer" containerID="b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"
Oct 31 04:29:01.960658 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.960636    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\") pod \"5eb03491-33e0-4f12-b262-7042c4e58958\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") "
Oct 31 04:29:01.960787 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:01.960651    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c\": container with ID starting with b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c not found: ID does not exist" containerID="b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"
Oct 31 04:29:01.960787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.960702    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c"} err="failed to get container status \"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c\": rpc error: code = NotFound desc = could not find container \"b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c\": container with ID starting with b0b3a7a4e66de0f11c3fbc23410719eb589621f139d06b1632d84492ab50ef2c not found: ID does not exist"
Oct 31 04:29:01.960787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.960726    5831 scope.go:122] "RemoveContainer" containerID="c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"
Oct 31 04:29:01.960787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.960752    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\" (UniqueName: \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\") pod \"5eb03491-33e0-4f12-b262-7042c4e58958\" (UID: \"5eb03491-33e0-4f12-b262-7042c4e58958\") "
Oct 31 04:29:01.961045 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.961017    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljhvf\" (UniqueName: \"kubernetes.io/projected/cb3f1f8f-18ae-452a-9554-3bc17d7c5551-kube-api-access-ljhvf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:01.961197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.961175    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume" pod "5eb03491-33e0-4f12-b262-7042c4e58958" (UID: "5eb03491-33e0-4f12-b262-7042c4e58958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:01.963583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.963266    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6" pod "5eb03491-33e0-4f12-b262-7042c4e58958" (UID: "5eb03491-33e0-4f12-b262-7042c4e58958"). InnerVolumeSpecName "kube-api-access-9rtd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:01.976934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.976810    5831 scope.go:122] "RemoveContainer" containerID="c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"
Oct 31 04:29:01.977200 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:01.977178    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242\": container with ID starting with c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242 not found: ID does not exist" containerID="c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"
Oct 31 04:29:01.977306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:01.977210    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242"} err="failed to get container status \"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242\": rpc error: code = NotFound desc = could not find container \"c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242\": container with ID starting with c2c9ceb8acf1955fa1961790cfa8956a9a111990a050c28239713d5ad9b4b242 not found: ID does not exist"
Oct 31 04:29:02.062332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.062281    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb03491-33e0-4f12-b262-7042c4e58958-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:02.062332 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.062318    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtd6\" (UniqueName: \"kubernetes.io/projected/5eb03491-33e0-4f12-b262-7042c4e58958-kube-api-access-9rtd6\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:02.169162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.169126    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 04:29:02.174445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.174413    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9360eb-7cc4854c7-8jmvh"]
Oct 31 04:29:02.355094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.355009    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 04:29:02.360257 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.360218    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9360eb-8548f6fb95-lwzhx"]
Oct 31 04:29:02.817613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.817578    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 31 04:29:02.818101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.817638    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 31 04:29:02.959176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:02.959144    5831 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" secret="" err="secret \"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\" not found"
Oct 31 04:29:03.817953 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:03.817917    5831 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 31 04:29:03.818359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:03.817978    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 31 04:29:03.965325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:03.965285    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy" containerID="cri-o://27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c" gracePeriod=30
Oct 31 04:29:04.133697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:04.133616    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" path="/var/lib/kubelet/pods/5eb03491-33e0-4f12-b262-7042c4e58958/volumes"
Oct 31 04:29:04.134047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:04.134033    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" path="/var/lib/kubelet/pods/cb3f1f8f-18ae-452a-9554-3bc17d7c5551/volumes"
Oct 31 04:29:06.059040 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:06.059001    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:29:11.268101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.268066    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d8444996-2rgdr_12c09a2b-78b4-4175-826f-781805aee163/authorino/0.log"
Oct 31 04:29:11.642824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.642782    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dwqhg_b13d7e68-3328-4ef1-9449-fddfcce22cea/manager/0.log"
Oct 31 04:29:11.726445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.726414    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:29:11.780200 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780166    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780225    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780255    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780275    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780300    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780344    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780409    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780441    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780507    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\") pod \"3fef98c1-865b-457f-baa2-990c4fd7532b\" (UID: \"3fef98c1-865b-457f-baa2-990c4fd7532b\") "
Oct 31 04:29:11.780726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780486    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:11.780726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780602    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:11.780997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780738    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:11.780997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780864    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.780997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780886    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.780997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.780901    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fef98c1-865b-457f-baa2-990c4fd7532b-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.781259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.781055    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:11.781397 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.781361    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:11.783075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.783032    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:29:11.783288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.783263    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:29:11.783473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.783453    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "kube-api-access-wdcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:11.783795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.783771    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token" pod "3fef98c1-865b-457f-baa2-990c4fd7532b" (UID: "3fef98c1-865b-457f-baa2-990c4fd7532b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881332    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881360    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881372    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881381    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcmz\" (UniqueName: \"kubernetes.io/projected/3fef98c1-865b-457f-baa2-990c4fd7532b-kube-api-access-wdcmz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881393    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.881414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.881402    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fef98c1-865b-457f-baa2-990c4fd7532b-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:29:11.893248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:11.893214    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mxvkt_7da6fd01-0c72-4691-b956-6807e32461c1/manager/0.log"
Oct 31 04:29:12.012791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.012759    5831 generic.go:356] "Generic (PLEG): container finished" podID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerID="27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c" exitCode=0
Oct 31 04:29:12.013011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.012821    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"
Oct 31 04:29:12.013011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.012836    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" event={"ID":"3fef98c1-865b-457f-baa2-990c4fd7532b","Type":"ContainerDied","Data":"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"}
Oct 31 04:29:12.013011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.012880    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq" event={"ID":"3fef98c1-865b-457f-baa2-990c4fd7532b","Type":"ContainerDied","Data":"53447ab70f280ad3dd737c45e9dbc449890248718fc2e80ab89b75e5fc3d0eac"}
Oct 31 04:29:12.013011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.012901    5831 scope.go:122] "RemoveContainer" containerID="27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"
Oct 31 04:29:12.023931 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.023911    5831 scope.go:122] "RemoveContainer" containerID="27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"
Oct 31 04:29:12.024225 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:12.024202    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c\": container with ID starting with 27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c not found: ID does not exist" containerID="27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"
Oct 31 04:29:12.024269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.024238    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c"} err="failed to get container status \"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c\": rpc error: code = NotFound desc = could not find container \"27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c\": container with ID starting with 27f41b68939015c5e9c47f39d7504f0e4fbca4e514fa5b350db9dd14271dd97c not found: ID does not exist"
Oct 31 04:29:12.164850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.164766    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:29:12.170604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:12.170575    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hwhwq"]
Oct 31 04:29:12.978737 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:12.978696    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:29:13.814500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:13.814460    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ss2vc_3778b1c5-f1c0-43d6-b45c-c121a45904d2/manager/0.log"
Oct 31 04:29:14.098225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:14.098149    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7phrx_2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc/limitador/0.log"
Oct 31 04:29:14.133348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:14.133320    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" path="/var/lib/kubelet/pods/3fef98c1-865b-457f-baa2-990c4fd7532b/volumes"
Oct 31 04:29:15.939273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:15.939242    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:29:16.659418 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:16.659376    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-k59sv_f93c4102-0f0a-401f-90f4-dc7d22955787/manager/0.log"
Oct 31 04:29:22.801161 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801122    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:29:22.801596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801578    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy"
Oct 31 04:29:22.801596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801593    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy"
Oct 31 04:29:22.801684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801605    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing"
Oct 31 04:29:22.801684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801610    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing"
Oct 31 04:29:22.801684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801622    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing"
Oct 31 04:29:22.801684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801627    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing"
Oct 31 04:29:22.801826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801698    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fef98c1-865b-457f-baa2-990c4fd7532b" containerName="istio-proxy"
Oct 31 04:29:22.801826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801706    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb03491-33e0-4f12-b262-7042c4e58958" containerName="payload-pre-processing"
Oct 31 04:29:22.801826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.801714    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3f1f8f-18ae-452a-9554-3bc17d7c5551" containerName="payload-processing"
Oct 31 04:29:22.806461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.806435    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.855787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.855750    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e863e405-openshift-default-dockercfg-jg5vn\""
Oct 31 04:29:22.876220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876181    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876237    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfh5v\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876263    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876391    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876455    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876549    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876587    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876619    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.876946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.876719    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.903563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.903502    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:29:22.977276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977237    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977292    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977458    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfh5v\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977493    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977551    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977604    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977649    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977685    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.977716 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977712    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.978030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.977972    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.978030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.978011    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.978149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.978133    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.978225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.978202    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.978370 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.978343    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.979833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.979809    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:22.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:22.980921    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:23.034994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.034962    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:23.035217 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.035196    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfh5v\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\") pod \"e2e-ait-e863e405-openshift-default-678bb86885-cv9bb\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") " pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:23.119346 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.119248    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:23.541348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.541313    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:29:23.545603 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:29:23.545572    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62f547b_1768_4955_a27b_f5185b55c238.slice/crio-9ac5c7b1359d232496294de37edbcad7f680832c7515fe141b0d5d14c2eb04fe WatchSource:0}: Error finding container 9ac5c7b1359d232496294de37edbcad7f680832c7515fe141b0d5d14c2eb04fe: Status 404 returned error can't find the container with id 9ac5c7b1359d232496294de37edbcad7f680832c7515fe141b0d5d14c2eb04fe
Oct 31 04:29:23.547753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.547719    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:23.547866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.547791    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:23.547866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:23.547821    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:24.095866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:24.095830    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerStarted","Data":"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"}
Oct 31 04:29:24.096267 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:24.095906    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerStarted","Data":"9ac5c7b1359d232496294de37edbcad7f680832c7515fe141b0d5d14c2eb04fe"}
Oct 31 04:29:24.119624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:24.119592    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:24.124924 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:24.124884    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:24.206323 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:24.206271    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podStartSLOduration=2.206257637 podStartE2EDuration="2.206257637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:24.204585935 +0000 UTC m=+4914.219147648" watchObservedRunningTime="2026-10-31 04:29:24.206257637 +0000 UTC m=+4914.220819409"
Oct 31 04:29:25.102853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:25.102813    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:25.104070 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:25.104046    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:29.956316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:29.956283    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:29:29.960453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:29.960427    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.043324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.043284    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.043845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.043386    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggz4n\" (UniqueName: \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.056440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.056380    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e863e405\""
Oct 31 04:29:30.056680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.056383    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e863e405-dockercfg-7gsk9\""
Oct 31 04:29:30.060203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.060176    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:29:30.108034 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.107991    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:29:30.114419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.114394    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.144277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.144245    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.144461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.144307    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggz4n\" (UniqueName: \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.145052 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.145013    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.160134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.160105    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:29:30.197149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.197114    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggz4n\" (UniqueName: \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\") pod \"payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.244919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.244827    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.245104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.245080    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk88\" (UniqueName: \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.320890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.320858    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e863e405-dockercfg-7gsk9\""
Oct 31 04:29:30.323978 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.323959    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:30.345730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.345698    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.345876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.345781    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk88\" (UniqueName: \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.346374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.346350    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.402567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.399484    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk88\" (UniqueName: \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\") pod \"payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") " pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.427812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.427785    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:30.697425 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.697397    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:29:30.699638 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:29:30.699607    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3454b88a_7b53_47e6_869a_55841e1182e8.slice/crio-ec61ba5d76f6844b22498b2ea8c824b793e8c155725cf88dca8d8fd40db95929 WatchSource:0}: Error finding container ec61ba5d76f6844b22498b2ea8c824b793e8c155725cf88dca8d8fd40db95929: Status 404 returned error can't find the container with id ec61ba5d76f6844b22498b2ea8c824b793e8c155725cf88dca8d8fd40db95929
Oct 31 04:29:30.797676 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:30.797648    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:29:30.799816 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:29:30.799789    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134bef05_cec6_43eb_98e2_e5215043a18a.slice/crio-229a600644168dcb7071e59c4f831b6bcf2993de3386fd29b1bae5a48b176f78 WatchSource:0}: Error finding container 229a600644168dcb7071e59c4f831b6bcf2993de3386fd29b1bae5a48b176f78: Status 404 returned error can't find the container with id 229a600644168dcb7071e59c4f831b6bcf2993de3386fd29b1bae5a48b176f78
Oct 31 04:29:31.144558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.144422    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" event={"ID":"134bef05-cec6-43eb-98e2-e5215043a18a","Type":"ContainerStarted","Data":"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"}
Oct 31 04:29:31.144558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.144486    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" event={"ID":"134bef05-cec6-43eb-98e2-e5215043a18a","Type":"ContainerStarted","Data":"229a600644168dcb7071e59c4f831b6bcf2993de3386fd29b1bae5a48b176f78"}
Oct 31 04:29:31.145103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.144604    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:31.146131 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.146103    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" event={"ID":"3454b88a-7b53-47e6-869a-55841e1182e8","Type":"ContainerStarted","Data":"bef904baf74549980f38a2e00c6e92770f09d6cef3c409d33694084cf498b7cf"}
Oct 31 04:29:31.146237 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.146152    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" event={"ID":"3454b88a-7b53-47e6-869a-55841e1182e8","Type":"ContainerStarted","Data":"ec61ba5d76f6844b22498b2ea8c824b793e8c155725cf88dca8d8fd40db95929"}
Oct 31 04:29:31.146237 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.146179    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:31.253355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.253297    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" podStartSLOduration=1.253283129 podStartE2EDuration="1.253283129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:31.251113804 +0000 UTC m=+4921.265675512" watchObservedRunningTime="2026-10-31 04:29:31.253283129 +0000 UTC m=+4921.267844836"
Oct 31 04:29:31.358127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:31.358056    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" podStartSLOduration=2.358038027 podStartE2EDuration="2.358038027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:31.355890214 +0000 UTC m=+4921.370451923" watchObservedRunningTime="2026-10-31 04:29:31.358038027 +0000 UTC m=+4921.372599742"
Oct 31 04:29:32.556239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.556202    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:29:32.560107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.560083    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.610422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.610386    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e863e405-serving-cert\""
Oct 31 04:29:32.610602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.610492    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:29:32.610602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.610564    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:29:32.656618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.656587    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:29:32.656777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.656673    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 04:29:32.658908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.658885    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:29:32.667143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.667120    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.667206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.667178    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.667377 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.667360    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.667424 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.667399    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsgb\" (UniqueName: \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.768524 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.768486    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.768754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.768582    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.768754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.768654    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.768754 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:32.768665    5831 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-serving-cert: secret "maas-api-e2e-ait-e863e405-serving-cert" not found
Oct 31 04:29:32.768754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.768689    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsgb\" (UniqueName: \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.768754 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:32.768735    5831 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls podName:93b3bee7-eaf8-4348-a8d0-f8381b6885bc nodeName:}" failed. No retries permitted until 2026-10-31 04:29:33.26871465 +0000 UTC m=+4923.283276348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls") pod "maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" (UID: "93b3bee7-eaf8-4348-a8d0-f8381b6885bc") : secret "maas-api-e2e-ait-e863e405-serving-cert" not found
Oct 31 04:29:32.769277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.769247    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.771304 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.771279    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:32.822273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:32.822198    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsgb\" (UniqueName: \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:33.273607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:33.273576    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:33.276301 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:33.276276    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") pod \"maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:33.474226 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:33.474191    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:33.854187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:33.854160    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:29:33.856904 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:29:33.856870    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b3bee7_eaf8_4348_a8d0_f8381b6885bc.slice/crio-914c7a2c8f80a42af0f581280d30c58bdcb5e9018d52d280ee098b04d75f3095 WatchSource:0}: Error finding container 914c7a2c8f80a42af0f581280d30c58bdcb5e9018d52d280ee098b04d75f3095: Status 404 returned error can't find the container with id 914c7a2c8f80a42af0f581280d30c58bdcb5e9018d52d280ee098b04d75f3095
Oct 31 04:29:33.858565 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:33.858517    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:29:34.166351 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:34.166265    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" event={"ID":"93b3bee7-eaf8-4348-a8d0-f8381b6885bc","Type":"ContainerStarted","Data":"914c7a2c8f80a42af0f581280d30c58bdcb5e9018d52d280ee098b04d75f3095"}
Oct 31 04:29:35.175243 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:35.175153    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" event={"ID":"93b3bee7-eaf8-4348-a8d0-f8381b6885bc","Type":"ContainerStarted","Data":"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"}
Oct 31 04:29:35.175662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:35.175290    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:35.282836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:35.282785    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" podStartSLOduration=2.250787272 podStartE2EDuration="3.282772057s" totalImagesPullingTime="1.031984785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:35.281180973 +0000 UTC m=+4925.295742693" watchObservedRunningTime="2026-10-31 04:29:35.282772057 +0000 UTC m=+4925.297333764"
Oct 31 04:29:40.875136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:40.875094    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:29:41.186158 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:41.186126    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:29:42.156832 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:42.156804    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:29:42.157351 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:42.156910    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:29:42.982560 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:29:42.978648    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:29:51.283606 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.283571    5831 generic.go:356] "Generic (PLEG): container finished" podID="c62f547b-1768-4955-a27b-f5185b55c238" containerID="326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51" exitCode=0
Oct 31 04:29:51.284094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.283640    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerDied","Data":"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"}
Oct 31 04:29:51.284094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.284034    5831 scope.go:122] "RemoveContainer" containerID="326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"
Oct 31 04:29:51.285220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.285190    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:51.285369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.285272    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:51.285369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:51.285344    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:29:52.315839 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:52.315797    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerStarted","Data":"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"}
Oct 31 04:29:53.120029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:53.119997    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:53.121324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:53.121292    5831 patch_prober.go:29] interesting pod/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 31 04:29:53.121470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:53.121353    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 31 04:29:54.120075 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:54.120037    5831 patch_prober.go:29] interesting pod/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 31 04:29:54.120553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:54.120092    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 31 04:29:55.103679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:55.103627    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:55.119768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:55.119733    5831 patch_prober.go:29] interesting pod/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 31 04:29:55.119930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:55.119800    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 31 04:29:56.120056 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:56.120017    5831 patch_prober.go:29] interesting pod/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 31 04:29:56.120576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:56.120073    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 31 04:29:57.119939 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:57.119901    5831 patch_prober.go:29] interesting pod/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 31 04:29:57.120141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:57.119959    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 31 04:29:58.124511 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:58.124477    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:29:58.125634 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:29:58.125616    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:30:02.674714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.674675    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"]
Oct 31 04:30:02.678430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.678407    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.710387 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.710357    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8f5\" (UniqueName: \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.710550 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.710401    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.710599 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.710551    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.728350 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.728320    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:30:02.774459 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.774425    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:30:02.776833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.776810    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"]
Oct 31 04:30:02.811142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.811106    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.811328 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.811167    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8f5\" (UniqueName: \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.811328 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.811195    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.811785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.811753    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.813721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.813703    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.864119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.864083    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8f5\" (UniqueName: \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\") pod \"maas-api-revoke-keys-e2e-ait-e863e405-9dcdm\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:02.992062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:02.992013    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:03.369880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:03.369847    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"]
Oct 31 04:30:03.373312 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:30:03.373281    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a1a759_dd76_4b72_9048_d388ec9cbdd8.slice/crio-9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b WatchSource:0}: Error finding container 9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b: Status 404 returned error can't find the container with id 9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b
Oct 31 04:30:03.383125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:03.383093    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm" event={"ID":"02a1a759-dd76-4b72-9048-d388ec9cbdd8","Type":"ContainerStarted","Data":"9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b"}
Oct 31 04:30:04.391047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:04.391010    5831 generic.go:356] "Generic (PLEG): container finished" podID="02a1a759-dd76-4b72-9048-d388ec9cbdd8" containerID="fd6e968cffe3dbf8bba07a0eb9fd46c7b61783f6c7d4dd894c271fbffbcb7130" exitCode=0
Oct 31 04:30:04.391511 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:04.391092    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm" event={"ID":"02a1a759-dd76-4b72-9048-d388ec9cbdd8","Type":"ContainerDied","Data":"fd6e968cffe3dbf8bba07a0eb9fd46c7b61783f6c7d4dd894c271fbffbcb7130"}
Oct 31 04:30:05.532855 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.532826    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:05.637476 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.637445    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\") pod \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") "
Oct 31 04:30:05.637721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.637552    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\" (UniqueName: \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\") pod \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") "
Oct 31 04:30:05.637721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.637657    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\") pod \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\" (UID: \"02a1a759-dd76-4b72-9048-d388ec9cbdd8\") "
Oct 31 04:30:05.638251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.638085    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca" pod "02a1a759-dd76-4b72-9048-d388ec9cbdd8" (UID: "02a1a759-dd76-4b72-9048-d388ec9cbdd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:05.640404 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.640373    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5" pod "02a1a759-dd76-4b72-9048-d388ec9cbdd8" (UID: "02a1a759-dd76-4b72-9048-d388ec9cbdd8"). InnerVolumeSpecName "kube-api-access-xl8f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:05.640404 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.640390    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca" pod "02a1a759-dd76-4b72-9048-d388ec9cbdd8" (UID: "02a1a759-dd76-4b72-9048-d388ec9cbdd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:05.739375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.739340    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl8f5\" (UniqueName: \"kubernetes.io/projected/02a1a759-dd76-4b72-9048-d388ec9cbdd8-kube-api-access-xl8f5\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:05.739375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.739367    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:05.739375 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:05.739377    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a1a759-dd76-4b72-9048-d388ec9cbdd8-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:06.403512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:06.403425    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"
Oct 31 04:30:06.403705 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:06.403421    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm" event={"ID":"02a1a759-dd76-4b72-9048-d388ec9cbdd8","Type":"ContainerDied","Data":"9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b"}
Oct 31 04:30:06.403705 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:06.403570    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6d41127ae13d52c7eaf29d95c35725d578cee65749c42f02245977247e9c2b"
Oct 31 04:30:12.978858 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:12.978817    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:30:14.186059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:14.186018    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:30:14.292499 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:14.292463    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:30:19.186571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.186501    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" podUID="134bef05-cec6-43eb-98e2-e5215043a18a" containerName="payload-processing" containerID="cri-o://f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061" gracePeriod=25
Oct 31 04:30:19.293230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.293169    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" podUID="3454b88a-7b53-47e6-869a-55841e1182e8" containerName="payload-pre-processing" containerID="cri-o://bef904baf74549980f38a2e00c6e92770f09d6cef3c409d33694084cf498b7cf" gracePeriod=25
Oct 31 04:30:19.444993 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.444910    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:30:19.482158 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.482126    5831 generic.go:356] "Generic (PLEG): container finished" podID="134bef05-cec6-43eb-98e2-e5215043a18a" containerID="f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061" exitCode=0
Oct 31 04:30:19.482298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.482179    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"
Oct 31 04:30:19.482298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.482208    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" event={"ID":"134bef05-cec6-43eb-98e2-e5215043a18a","Type":"ContainerDied","Data":"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"}
Oct 31 04:30:19.482298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.482277    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq" event={"ID":"134bef05-cec6-43eb-98e2-e5215043a18a","Type":"ContainerDied","Data":"229a600644168dcb7071e59c4f831b6bcf2993de3386fd29b1bae5a48b176f78"}
Oct 31 04:30:19.482461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.482312    5831 scope.go:122] "RemoveContainer" containerID="f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"
Oct 31 04:30:19.484667 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.484638    5831 generic.go:356] "Generic (PLEG): container finished" podID="3454b88a-7b53-47e6-869a-55841e1182e8" containerID="bef904baf74549980f38a2e00c6e92770f09d6cef3c409d33694084cf498b7cf" exitCode=0
Oct 31 04:30:19.484784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.484673    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" event={"ID":"3454b88a-7b53-47e6-869a-55841e1182e8","Type":"ContainerDied","Data":"bef904baf74549980f38a2e00c6e92770f09d6cef3c409d33694084cf498b7cf"}
Oct 31 04:30:19.493363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.493275    5831 scope.go:122] "RemoveContainer" containerID="f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"
Oct 31 04:30:19.493639 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:19.493616    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061\": container with ID starting with f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061 not found: ID does not exist" containerID="f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"
Oct 31 04:30:19.493775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.493646    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061"} err="failed to get container status \"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061\": rpc error: code = NotFound desc = could not find container \"f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061\": container with ID starting with f1364df2adf06af2295a5e5c05e2fac3e661b9a9a7f0a8c82c595663e6617061 not found: ID does not exist"
Oct 31 04:30:19.551596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.551573    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:30:19.577393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.577354    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\" (UniqueName: \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\") pod \"134bef05-cec6-43eb-98e2-e5215043a18a\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") "
Oct 31 04:30:19.577580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.577419    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\") pod \"134bef05-cec6-43eb-98e2-e5215043a18a\" (UID: \"134bef05-cec6-43eb-98e2-e5215043a18a\") "
Oct 31 04:30:19.577844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.577816    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume" pod "134bef05-cec6-43eb-98e2-e5215043a18a" (UID: "134bef05-cec6-43eb-98e2-e5215043a18a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:19.579839 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.579812    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88" pod "134bef05-cec6-43eb-98e2-e5215043a18a" (UID: "134bef05-cec6-43eb-98e2-e5215043a18a"). InnerVolumeSpecName "kube-api-access-hnk88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:19.678303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.678270    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\" (UniqueName: \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\") pod \"3454b88a-7b53-47e6-869a-55841e1182e8\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") "
Oct 31 04:30:19.678491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.678321    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\") pod \"3454b88a-7b53-47e6-869a-55841e1182e8\" (UID: \"3454b88a-7b53-47e6-869a-55841e1182e8\") "
Oct 31 04:30:19.678726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.678703    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume" pod "3454b88a-7b53-47e6-869a-55841e1182e8" (UID: "3454b88a-7b53-47e6-869a-55841e1182e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:19.678770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.678737    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnk88\" (UniqueName: \"kubernetes.io/projected/134bef05-cec6-43eb-98e2-e5215043a18a-kube-api-access-hnk88\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:19.678770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.678760    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134bef05-cec6-43eb-98e2-e5215043a18a-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:19.680668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.680650    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n" pod "3454b88a-7b53-47e6-869a-55841e1182e8" (UID: "3454b88a-7b53-47e6-869a-55841e1182e8"). InnerVolumeSpecName "kube-api-access-ggz4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:19.780006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.779924    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggz4n\" (UniqueName: \"kubernetes.io/projected/3454b88a-7b53-47e6-869a-55841e1182e8-kube-api-access-ggz4n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:19.780006 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.779952    5831 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3454b88a-7b53-47e6-869a-55841e1182e8-plugins-config-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:19.892788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.892755    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:30:19.897000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:19.896965    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e863e405-b4bdcb495-55jgq"]
Oct 31 04:30:20.133288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.133212    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134bef05-cec6-43eb-98e2-e5215043a18a" path="/var/lib/kubelet/pods/134bef05-cec6-43eb-98e2-e5215043a18a/volumes"
Oct 31 04:30:20.493252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.493215    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb" event={"ID":"3454b88a-7b53-47e6-869a-55841e1182e8","Type":"ContainerDied","Data":"ec61ba5d76f6844b22498b2ea8c824b793e8c155725cf88dca8d8fd40db95929"}
Oct 31 04:30:20.493252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.493240    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"
Oct 31 04:30:20.493848 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.493272    5831 scope.go:122] "RemoveContainer" containerID="bef904baf74549980f38a2e00c6e92770f09d6cef3c409d33694084cf498b7cf"
Oct 31 04:30:20.644263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.644226    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:30:20.690203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:20.690166    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e863e405-6c6bdc45c9-hr6hb"]
Oct 31 04:30:21.452972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.452944    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:30:21.453178 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.453132    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" podUID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" containerName="maas-api" containerID="cri-o://3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17" gracePeriod=30
Oct 31 04:30:21.713455 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.713396    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:30:21.799987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.799948    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\") pod \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") "
Oct 31 04:30:21.800177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.800008    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\" (UniqueName: \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\") pod \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") "
Oct 31 04:30:21.800177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.800039    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") pod \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") "
Oct 31 04:30:21.800177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.800070    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\") pod \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\" (UID: \"93b3bee7-eaf8-4348-a8d0-f8381b6885bc\") "
Oct 31 04:30:21.800513 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.800484    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca" pod "93b3bee7-eaf8-4348-a8d0-f8381b6885bc" (UID: "93b3bee7-eaf8-4348-a8d0-f8381b6885bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:21.802380 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.802348    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs" pod "93b3bee7-eaf8-4348-a8d0-f8381b6885bc" (UID: "93b3bee7-eaf8-4348-a8d0-f8381b6885bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:21.802489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.802382    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb" pod "93b3bee7-eaf8-4348-a8d0-f8381b6885bc" (UID: "93b3bee7-eaf8-4348-a8d0-f8381b6885bc"). InnerVolumeSpecName "kube-api-access-sbsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:21.802489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.802401    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls" pod "93b3bee7-eaf8-4348-a8d0-f8381b6885bc" (UID: "93b3bee7-eaf8-4348-a8d0-f8381b6885bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:21.901247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.901214    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:21.901247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.901246    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsgb\" (UniqueName: \"kubernetes.io/projected/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-kube-api-access-sbsgb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:21.901486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.901261    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:21.901486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:21.901273    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b3bee7-eaf8-4348-a8d0-f8381b6885bc-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:22.134974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.134896    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3454b88a-7b53-47e6-869a-55841e1182e8" path="/var/lib/kubelet/pods/3454b88a-7b53-47e6-869a-55841e1182e8/volumes"
Oct 31 04:30:22.509752 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.509714    5831 generic.go:356] "Generic (PLEG): container finished" podID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" containerID="3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17" exitCode=0
Oct 31 04:30:22.509956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.509782    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"
Oct 31 04:30:22.509956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.509806    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" event={"ID":"93b3bee7-eaf8-4348-a8d0-f8381b6885bc","Type":"ContainerDied","Data":"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"}
Oct 31 04:30:22.509956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.509860    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz" event={"ID":"93b3bee7-eaf8-4348-a8d0-f8381b6885bc","Type":"ContainerDied","Data":"914c7a2c8f80a42af0f581280d30c58bdcb5e9018d52d280ee098b04d75f3095"}
Oct 31 04:30:22.509956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.509885    5831 scope.go:122] "RemoveContainer" containerID="3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"
Oct 31 04:30:22.520522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.520502    5831 scope.go:122] "RemoveContainer" containerID="3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"
Oct 31 04:30:22.520869 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:22.520845    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17\": container with ID starting with 3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17 not found: ID does not exist" containerID="3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"
Oct 31 04:30:22.520938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.520879    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17"} err="failed to get container status \"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17\": rpc error: code = NotFound desc = could not find container \"3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17\": container with ID starting with 3cfc5e2994686afb05c314858618e626a11c4f8da1583559b8a4e4392d9f5b17 not found: ID does not exist"
Oct 31 04:30:22.662595 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.662550    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:30:22.707056 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:22.707024    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e863e405-5859cc97cf-xz6pz"]
Oct 31 04:30:23.562714 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:23.562673    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"]
Oct 31 04:30:23.564402 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:23.564377    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e863e405-9dcdm"]
Oct 31 04:30:24.133491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:24.133455    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a1a759-dd76-4b72-9048-d388ec9cbdd8" path="/var/lib/kubelet/pods/02a1a759-dd76-4b72-9048-d388ec9cbdd8/volumes"
Oct 31 04:30:24.133909 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:24.133894    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" path="/var/lib/kubelet/pods/93b3bee7-eaf8-4348-a8d0-f8381b6885bc/volumes"
Oct 31 04:30:25.993210 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:25.993176    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:30:28.791309 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:28.791275    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:30:28.791897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:28.791547    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy" containerID="cri-o://b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8" gracePeriod=30
Oct 31 04:30:33.145955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.145917    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 04:30:33.146408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.146174    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7c946769f6-k59sv" podUID="f93c4102-0f0a-401f-90f4-dc7d22955787" containerName="manager" containerID="cri-o://2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915" gracePeriod=30
Oct 31 04:30:33.415308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.415279    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 04:30:33.510852 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.510816    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\" (UniqueName: \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\") pod \"f93c4102-0f0a-401f-90f4-dc7d22955787\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") "
Oct 31 04:30:33.511030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.510898    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\") pod \"f93c4102-0f0a-401f-90f4-dc7d22955787\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") "
Oct 31 04:30:33.511030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.511022    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\") pod \"f93c4102-0f0a-401f-90f4-dc7d22955787\" (UID: \"f93c4102-0f0a-401f-90f4-dc7d22955787\") "
Oct 31 04:30:33.513269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.513242    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs" pod "f93c4102-0f0a-401f-90f4-dc7d22955787" (UID: "f93c4102-0f0a-401f-90f4-dc7d22955787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:33.513418 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.513278    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert" pod "f93c4102-0f0a-401f-90f4-dc7d22955787" (UID: "f93c4102-0f0a-401f-90f4-dc7d22955787"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:33.513418 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.513302    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc" pod "f93c4102-0f0a-401f-90f4-dc7d22955787" (UID: "f93c4102-0f0a-401f-90f4-dc7d22955787"). InnerVolumeSpecName "kube-api-access-rj7wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:33.579750 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.579713    5831 generic.go:356] "Generic (PLEG): container finished" podID="f93c4102-0f0a-401f-90f4-dc7d22955787" containerID="2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915" exitCode=0
Oct 31 04:30:33.579958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.579764    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-k59sv"
Oct 31 04:30:33.579958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.579777    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-k59sv" event={"ID":"f93c4102-0f0a-401f-90f4-dc7d22955787","Type":"ContainerDied","Data":"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"}
Oct 31 04:30:33.579958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.579832    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-k59sv" event={"ID":"f93c4102-0f0a-401f-90f4-dc7d22955787","Type":"ContainerDied","Data":"c5100c16265405774cb10c204e1934e10247086e6e4eba73f1f8808f99eb1d4c"}
Oct 31 04:30:33.579958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.579856    5831 scope.go:122] "RemoveContainer" containerID="2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"
Oct 31 04:30:33.592943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.592922    5831 scope.go:122] "RemoveContainer" containerID="2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"
Oct 31 04:30:33.593364 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:33.593332    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915\": container with ID starting with 2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915 not found: ID does not exist" containerID="2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"
Oct 31 04:30:33.593491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.593377    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915"} err="failed to get container status \"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915\": rpc error: code = NotFound desc = could not find container \"2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915\": container with ID starting with 2c7e7f173e1170cba5834a5e317bdc38d49131bfa84586b25a8268bda0806915 not found: ID does not exist"
Oct 31 04:30:33.611788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.611760    5831 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-webhook-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:33.611788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.611784    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7wc\" (UniqueName: \"kubernetes.io/projected/f93c4102-0f0a-401f-90f4-dc7d22955787-kube-api-access-rj7wc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:33.611788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.611797    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f93c4102-0f0a-401f-90f4-dc7d22955787-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:33.733767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.733726    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 04:30:33.741432 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:33.741401    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c946769f6-k59sv"]
Oct 31 04:30:34.058272 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.058243    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:30:34.116653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116621    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116669    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116690    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116707    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116750    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116775    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.116850 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116818    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.117225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.116861    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.117225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117027    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:34.117225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117154    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.117225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117183    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:34.117660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117305    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:34.117660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117360    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:34.117660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.117458    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:34.119107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.119080    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:34.119236 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.119218    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:30:34.119612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.119592    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:34.134660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.134629    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93c4102-0f0a-401f-90f4-dc7d22955787" path="/var/lib/kubelet/pods/f93c4102-0f0a-401f-90f4-dc7d22955787/volumes"
Oct 31 04:30:34.218325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218283    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\") pod \"c62f547b-1768-4955-a27b-f5185b55c238\" (UID: \"c62f547b-1768-4955-a27b-f5185b55c238\") "
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218485    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218506    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218521    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c62f547b-1768-4955-a27b-f5185b55c238-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218554    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c62f547b-1768-4955-a27b-f5185b55c238-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218568    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218579    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.218872 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.218592    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c62f547b-1768-4955-a27b-f5185b55c238-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.220668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.220641    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v" pod "c62f547b-1768-4955-a27b-f5185b55c238" (UID: "c62f547b-1768-4955-a27b-f5185b55c238"). InnerVolumeSpecName "kube-api-access-nfh5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:34.319305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.319266    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfh5v\" (UniqueName: \"kubernetes.io/projected/c62f547b-1768-4955-a27b-f5185b55c238-kube-api-access-nfh5v\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:34.570364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570279    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:30:34.570719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570704    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3454b88a-7b53-47e6-869a-55841e1182e8" containerName="payload-pre-processing"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570721    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3454b88a-7b53-47e6-869a-55841e1182e8" containerName="payload-pre-processing"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570734    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570739    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570752    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f93c4102-0f0a-401f-90f4-dc7d22955787" containerName="manager"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570757    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93c4102-0f0a-401f-90f4-dc7d22955787" containerName="manager"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570764    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570770    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570778    5831 cpu_manager.go:414] "containerMap: removing container" podUID="02a1a759-dd76-4b72-9048-d388ec9cbdd8" containerName="revoke-keys"
Oct 31 04:30:34.570779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570783    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a1a759-dd76-4b72-9048-d388ec9cbdd8" containerName="revoke-keys"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570792    5831 cpu_manager.go:414] "containerMap: removing container" podUID="134bef05-cec6-43eb-98e2-e5215043a18a" containerName="payload-processing"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570797    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134bef05-cec6-43eb-98e2-e5215043a18a" containerName="payload-processing"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570807    5831 cpu_manager.go:414] "containerMap: removing container" podUID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" containerName="maas-api"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570812    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" containerName="maas-api"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570876    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3454b88a-7b53-47e6-869a-55841e1182e8" containerName="payload-pre-processing"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570885    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570906    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93c4102-0f0a-401f-90f4-dc7d22955787" containerName="manager"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570912    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62f547b-1768-4955-a27b-f5185b55c238" containerName="istio-proxy"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570918    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a1a759-dd76-4b72-9048-d388ec9cbdd8" containerName="revoke-keys"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570925    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="134bef05-cec6-43eb-98e2-e5215043a18a" containerName="payload-processing"
Oct 31 04:30:34.571129 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.570932    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b3bee7-eaf8-4348-a8d0-f8381b6885bc" containerName="maas-api"
Oct 31 04:30:34.575601 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.575565    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.589152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.589119    5831 generic.go:356] "Generic (PLEG): container finished" podID="c62f547b-1768-4955-a27b-f5185b55c238" containerID="b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8" exitCode=0
Oct 31 04:30:34.589289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.589163    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"
Oct 31 04:30:34.589289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.589196    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerDied","Data":"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"}
Oct 31 04:30:34.589289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.589247    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb" event={"ID":"c62f547b-1768-4955-a27b-f5185b55c238","Type":"ContainerDied","Data":"9ac5c7b1359d232496294de37edbcad7f680832c7515fe141b0d5d14c2eb04fe"}
Oct 31 04:30:34.589289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.589272    5831 scope.go:122] "RemoveContainer" containerID="b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"
Oct 31 04:30:34.601114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.601095    5831 scope.go:122] "RemoveContainer" containerID="326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"
Oct 31 04:30:34.610485 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.610465    5831 scope.go:122] "RemoveContainer" containerID="b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"
Oct 31 04:30:34.610868 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:34.610851    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8\": container with ID starting with b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8 not found: ID does not exist" containerID="b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"
Oct 31 04:30:34.610931 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.610877    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8"} err="failed to get container status \"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8\": rpc error: code = NotFound desc = could not find container \"b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8\": container with ID starting with b6ca0b4668abf7551693f38b4bcea9c2988e212ed23f832e46b6a022329a1ec8 not found: ID does not exist"
Oct 31 04:30:34.610931 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.610898    5831 scope.go:122] "RemoveContainer" containerID="326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"
Oct 31 04:30:34.611157 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:34.611137    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51\": container with ID starting with 326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51 not found: ID does not exist" containerID="326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"
Oct 31 04:30:34.611226 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.611168    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51"} err="failed to get container status \"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51\": rpc error: code = NotFound desc = could not find container \"326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51\": container with ID starting with 326c486af4364308fea7f6647e4b110142fd19f2c34e6c9767b116f2136d4b51 not found: ID does not exist"
Oct 31 04:30:34.621745 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.621719    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.621870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.621822    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kld\" (UniqueName: \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.621929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.621869    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.625717 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.625695    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 04:30:34.625717 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.625691    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 04:30:34.625921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.625705    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 04:30:34.625921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.625708    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l2xw2\""
Oct 31 04:30:34.674141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.674108    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:30:34.722723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.722686    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.722904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.722761    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kld\" (UniqueName: \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.722904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.722779    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.725512 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.725481    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.725743 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.725720    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.776158 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.776126    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kld\" (UniqueName: \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\") pod \"maas-controller-7c946769f6-7vw7l\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") " pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:34.787337 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.787305    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:30:34.790683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.790655    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e863e405-openshift-default-678bb86885-cv9bb"]
Oct 31 04:30:34.889002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:34.888905    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:35.265374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:35.265344    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:30:35.269231 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:30:35.269201    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da5c0a_da6e_4381_b633_07e5d4a4955c.slice/crio-addc780b96ec93586e22a4d39a8387d7a4bb63ae4e256f3b9847723271c8ba44 WatchSource:0}: Error finding container addc780b96ec93586e22a4d39a8387d7a4bb63ae4e256f3b9847723271c8ba44: Status 404 returned error can't find the container with id addc780b96ec93586e22a4d39a8387d7a4bb63ae4e256f3b9847723271c8ba44
Oct 31 04:30:35.598738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:35.598635    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-7vw7l" event={"ID":"b5da5c0a-da6e-4381-b633-07e5d4a4955c","Type":"ContainerStarted","Data":"addc780b96ec93586e22a4d39a8387d7a4bb63ae4e256f3b9847723271c8ba44"}
Oct 31 04:30:36.135948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:36.135915    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62f547b-1768-4955-a27b-f5185b55c238" path="/var/lib/kubelet/pods/c62f547b-1768-4955-a27b-f5185b55c238/volumes"
Oct 31 04:30:36.607744 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:36.607697    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-7vw7l" event={"ID":"b5da5c0a-da6e-4381-b633-07e5d4a4955c","Type":"ContainerStarted","Data":"85e18c5694265dbab00c9eabc8db79cb2eaadc6f221e3a75ec8052686c1e6c5a"}
Oct 31 04:30:36.608193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:36.607905    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:36.717735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:36.717684    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7c946769f6-7vw7l" podStartSLOduration=1.6548104380000002 podStartE2EDuration="2.717669623s" totalImagesPullingTime="1.062859185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:36.716231297 +0000 UTC m=+4986.730793007" watchObservedRunningTime="2026-10-31 04:30:36.717669623 +0000 UTC m=+4986.732231365"
Oct 31 04:30:42.978157 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:30:42.978115    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:30:47.615974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:47.615937    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:30:57.021938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.021902    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"]
Oct 31 04:30:57.028709 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.028663    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.077757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.077716    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"]
Oct 31 04:30:57.121017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.120979    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:30:57.121017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.121004    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:30:57.121263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.121023    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:30:57.122881 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.122859    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:30:57.125172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.125147    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479wb\" (UniqueName: \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.125241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.125184    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.125241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.125203    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.226591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.226550    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479wb\" (UniqueName: \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.226806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.226636    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.226806 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.226667    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.227404 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.227376    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.229370 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.229351    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.279854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.279785    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479wb\" (UniqueName: \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\") pod \"maas-api-revoke-sub-85237e45-cwjv4\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.344235 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.344199    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:57.710259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.710230    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"]
Oct 31 04:30:57.713504 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:30:57.713472    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b86c892_984f_41c4_bc7c_30c0cee1127b.slice/crio-d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26 WatchSource:0}: Error finding container d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26: Status 404 returned error can't find the container with id d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26
Oct 31 04:30:57.741440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:57.741413    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4" event={"ID":"6b86c892-984f-41c4-bc7c-30c0cee1127b","Type":"ContainerStarted","Data":"d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26"}
Oct 31 04:30:58.748719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:58.748681    5831 generic.go:356] "Generic (PLEG): container finished" podID="6b86c892-984f-41c4-bc7c-30c0cee1127b" containerID="8656bc0ac24a5c098b30a8372469b748935efd5d9c3d534b1cdcdc97354655a3" exitCode=0
Oct 31 04:30:58.749136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:58.748772    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4" event={"ID":"6b86c892-984f-41c4-bc7c-30c0cee1127b","Type":"ContainerDied","Data":"8656bc0ac24a5c098b30a8372469b748935efd5d9c3d534b1cdcdc97354655a3"}
Oct 31 04:30:59.897493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.897470    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:30:59.953816 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.953787    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\" (UniqueName: \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\") pod \"6b86c892-984f-41c4-bc7c-30c0cee1127b\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") "
Oct 31 04:30:59.953982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.953850    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\") pod \"6b86c892-984f-41c4-bc7c-30c0cee1127b\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") "
Oct 31 04:30:59.953982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.953890    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\") pod \"6b86c892-984f-41c4-bc7c-30c0cee1127b\" (UID: \"6b86c892-984f-41c4-bc7c-30c0cee1127b\") "
Oct 31 04:30:59.954222 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.954188    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca" pod "6b86c892-984f-41c4-bc7c-30c0cee1127b" (UID: "6b86c892-984f-41c4-bc7c-30c0cee1127b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:59.956067 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.956046    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca" pod "6b86c892-984f-41c4-bc7c-30c0cee1127b" (UID: "6b86c892-984f-41c4-bc7c-30c0cee1127b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:59.956162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:30:59.956115    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb" pod "6b86c892-984f-41c4-bc7c-30c0cee1127b" (UID: "6b86c892-984f-41c4-bc7c-30c0cee1127b"). InnerVolumeSpecName "kube-api-access-479wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:31:00.055411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.055329    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479wb\" (UniqueName: \"kubernetes.io/projected/6b86c892-984f-41c4-bc7c-30c0cee1127b-kube-api-access-479wb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:00.055411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.055357    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:00.055411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.055369    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b86c892-984f-41c4-bc7c-30c0cee1127b-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:00.761884 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.761851    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"
Oct 31 04:31:00.762073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.761884    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4" event={"ID":"6b86c892-984f-41c4-bc7c-30c0cee1127b","Type":"ContainerDied","Data":"d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26"}
Oct 31 04:31:00.762073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:00.761946    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0de674910843f3a739c8463685c99642d56c1bf6b3165ebffc4a4c40a606c26"
Oct 31 04:31:07.668326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.668289    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"]
Oct 31 04:31:07.668770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.668688    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6b86c892-984f-41c4-bc7c-30c0cee1127b" containerName="revoke-keys"
Oct 31 04:31:07.668770 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.668700    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b86c892-984f-41c4-bc7c-30c0cee1127b" containerName="revoke-keys"
Oct 31 04:31:07.668848 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.668783    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b86c892-984f-41c4-bc7c-30c0cee1127b" containerName="revoke-keys"
Oct 31 04:31:07.671990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.671973    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.714183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.714150    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"]
Oct 31 04:31:07.717731 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.717716    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.721983 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.721957    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722009    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722070    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4kk\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-kube-api-access-dp4kk\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722161 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722135    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722161 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722153    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722176    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722280 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722251    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722362 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722343    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc914aa8-e974-404a-b1e4-3d5c05828693-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.722420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.722405    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.727227 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.727206    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 04:31:07.765741 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.765711    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"]
Oct 31 04:31:07.812420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.812383    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"]
Oct 31 04:31:07.823008 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.822976    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823018    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823149 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823065    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc914aa8-e974-404a-b1e4-3d5c05828693-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823226    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823266    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gt7\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-kube-api-access-x5gt7\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823308    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823386    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823474 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823450    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823492    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823516    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4kk\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-kube-api-access-dp4kk\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823595    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823602    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823621    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823655    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.823885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823678    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.823885 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823717    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.824002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823899    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.824002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823924    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.824002 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.823924    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc914aa8-e974-404a-b1e4-3d5c05828693-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.824171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.824006    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc00eba-a262-434c-8c79-d887a3cb9697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.824171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.824048    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.824171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.824079    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.824171 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.824111    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc00eba-a262-434c-8c79-d887a3cb9697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.826201 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.826178    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.826597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.826576    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.876090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.876050    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.876234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.876133    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4kk\" (UniqueName: \"kubernetes.io/projected/bc914aa8-e974-404a-b1e4-3d5c05828693-kube-api-access-dp4kk\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vrf\" (UID: \"bc914aa8-e974-404a-b1e4-3d5c05828693\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:07.924933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.924835    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.924933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.924885    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.924933 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.924912    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc00eba-a262-434c-8c79-d887a3cb9697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.924950    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc00eba-a262-434c-8c79-d887a3cb9697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.924981    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925012    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925078    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gt7\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-kube-api-access-x5gt7\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925113    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925147    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925336    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925734 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925707    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925784    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.925890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.925868    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.926095 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.926076    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc00eba-a262-434c-8c79-d887a3cb9697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.927837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.927818    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc00eba-a262-434c-8c79-d887a3cb9697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.927930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.927909    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc00eba-a262-434c-8c79-d887a3cb9697-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.977652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.977619    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.977817 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.977700    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gt7\" (UniqueName: \"kubernetes.io/projected/acc00eba-a262-434c-8c79-d887a3cb9697-kube-api-access-x5gt7\") pod \"maas-default-gateway-openshift-default-6886b96f9-nhf4g\" (UID: \"acc00eba-a262-434c-8c79-d887a3cb9697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:07.984679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:07.984661    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:08.030544 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.030491    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:08.399795 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.399758    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"]
Oct 31 04:31:08.439498 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:31:08.439471    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc914aa8_e974_404a_b1e4_3d5c05828693.slice/crio-dc4a4354d2120dfa1a987ed0536c590c9c5099524098b76f487200503559e5d1 WatchSource:0}: Error finding container dc4a4354d2120dfa1a987ed0536c590c9c5099524098b76f487200503559e5d1: Status 404 returned error can't find the container with id dc4a4354d2120dfa1a987ed0536c590c9c5099524098b76f487200503559e5d1
Oct 31 04:31:08.441688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.441654    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.441817 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.441734    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.441817 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.441792    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.451967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.451942    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"]
Oct 31 04:31:08.454374 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:31:08.454335    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc00eba_a262_434c_8c79_d887a3cb9697.slice/crio-8db90d3f938fdfacccfc87fadfe83192cf7ea6f49e53f2144349c2ad1bbf8a84 WatchSource:0}: Error finding container 8db90d3f938fdfacccfc87fadfe83192cf7ea6f49e53f2144349c2ad1bbf8a84: Status 404 returned error can't find the container with id 8db90d3f938fdfacccfc87fadfe83192cf7ea6f49e53f2144349c2ad1bbf8a84
Oct 31 04:31:08.456873 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.456839    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.456977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.456904    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.456977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.456932    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:08.814010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.813965    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" event={"ID":"bc914aa8-e974-404a-b1e4-3d5c05828693","Type":"ContainerStarted","Data":"7bca88a2caa04c64711c02f285e5a74642f8324796dd40bf867e235610e2aafc"}
Oct 31 04:31:08.814493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.814398    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" event={"ID":"bc914aa8-e974-404a-b1e4-3d5c05828693","Type":"ContainerStarted","Data":"dc4a4354d2120dfa1a987ed0536c590c9c5099524098b76f487200503559e5d1"}
Oct 31 04:31:08.816686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.816655    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" event={"ID":"acc00eba-a262-434c-8c79-d887a3cb9697","Type":"ContainerStarted","Data":"cdb657b4b3ead0bdb2a0f62fdba004a7706892a17cace97dcb1de4a020fc1d36"}
Oct 31 04:31:08.816837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.816719    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" event={"ID":"acc00eba-a262-434c-8c79-d887a3cb9697","Type":"ContainerStarted","Data":"8db90d3f938fdfacccfc87fadfe83192cf7ea6f49e53f2144349c2ad1bbf8a84"}
Oct 31 04:31:08.985676 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:08.985638    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:09.031108 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.031068    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:09.033316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.033281    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:09.033589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.033562    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:09.041115 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.041040    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podStartSLOduration=2.041021036 podStartE2EDuration="2.041021036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:09.037982905 +0000 UTC m=+5019.052544615" watchObservedRunningTime="2026-10-31 04:31:09.041021036 +0000 UTC m=+5019.055582745"
Oct 31 04:31:09.041927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.041893    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podStartSLOduration=2.041883418 podStartE2EDuration="2.041883418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:08.926235022 +0000 UTC m=+5018.940796730" watchObservedRunningTime="2026-10-31 04:31:09.041883418 +0000 UTC m=+5019.056445123"
Oct 31 04:31:09.539652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.539615    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:31:09.539857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.539816    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7c946769f6-7vw7l" podUID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" containerName="manager" containerID="cri-o://85e18c5694265dbab00c9eabc8db79cb2eaadc6f221e3a75ec8052686c1e6c5a" gracePeriod=30
Oct 31 04:31:09.848805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.847479    5831 generic.go:356] "Generic (PLEG): container finished" podID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" containerID="85e18c5694265dbab00c9eabc8db79cb2eaadc6f221e3a75ec8052686c1e6c5a" exitCode=0
Oct 31 04:31:09.848805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.848208    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-7vw7l" event={"ID":"b5da5c0a-da6e-4381-b633-07e5d4a4955c","Type":"ContainerDied","Data":"85e18c5694265dbab00c9eabc8db79cb2eaadc6f221e3a75ec8052686c1e6c5a"}
Oct 31 04:31:09.941636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:09.941607    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:31:10.015556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.014294    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:10.015556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.014360    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:10.035555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.034032    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:10.035555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.034091    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:10.047504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.047472    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\") pod \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") "
Oct 31 04:31:10.047698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.047553    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\") pod \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") "
Oct 31 04:31:10.047698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.047615    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\" (UniqueName: \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\") pod \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\" (UID: \"b5da5c0a-da6e-4381-b633-07e5d4a4955c\") "
Oct 31 04:31:10.050127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.049989    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert" pod "b5da5c0a-da6e-4381-b633-07e5d4a4955c" (UID: "b5da5c0a-da6e-4381-b633-07e5d4a4955c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:31:10.050127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.050036    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs" pod "b5da5c0a-da6e-4381-b633-07e5d4a4955c" (UID: "b5da5c0a-da6e-4381-b633-07e5d4a4955c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:31:10.050127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.050095    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld" pod "b5da5c0a-da6e-4381-b633-07e5d4a4955c" (UID: "b5da5c0a-da6e-4381-b633-07e5d4a4955c"). InnerVolumeSpecName "kube-api-access-p4kld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:31:10.148656 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.148580    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4kld\" (UniqueName: \"kubernetes.io/projected/b5da5c0a-da6e-4381-b633-07e5d4a4955c-kube-api-access-p4kld\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:10.148851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.148836    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:10.148928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.148919    5831 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5da5c0a-da6e-4381-b633-07e5d4a4955c-webhook-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:10.855497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.855469    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-7vw7l"
Oct 31 04:31:10.855926 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.855503    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-7vw7l" event={"ID":"b5da5c0a-da6e-4381-b633-07e5d4a4955c","Type":"ContainerDied","Data":"addc780b96ec93586e22a4d39a8387d7a4bb63ae4e256f3b9847723271c8ba44"}
Oct 31 04:31:10.855926 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.855570    5831 scope.go:122] "RemoveContainer" containerID="85e18c5694265dbab00c9eabc8db79cb2eaadc6f221e3a75ec8052686c1e6c5a"
Oct 31 04:31:10.998556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.986942    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:10.998556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:10.987005    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:11.028870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.028795    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:31:11.033997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.033962    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:11.034140 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.034029    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:11.070398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.070370    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c946769f6-7vw7l"]
Oct 31 04:31:11.986991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.986953    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:11.987615 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:11.987580    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:12.031823 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.031778    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:12.032147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.032115    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:12.136042 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.136004    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" path="/var/lib/kubelet/pods/b5da5c0a-da6e-4381-b633-07e5d4a4955c/volumes"
Oct 31 04:31:12.569094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.569034    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c946769f6-6pgn5"]
Oct 31 04:31:12.569621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.569607    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" containerName="manager"
Oct 31 04:31:12.569683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.569623    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" containerName="manager"
Oct 31 04:31:12.569726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.569707    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5da5c0a-da6e-4381-b633-07e5d4a4955c" containerName="manager"
Oct 31 04:31:12.574442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.574423    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.630044 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.630011    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 04:31:12.671774 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.671735    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-metrics-certs\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.671981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.671860    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96px\" (UniqueName: \"kubernetes.io/projected/90ef7101-8986-453d-aa53-e1a5c823b6cb-kube-api-access-s96px\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.671981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.671891    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-webhook-cert\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.676190 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.676150    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 04:31:12.676341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.676289    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 04:31:12.676453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.676414    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l2xw2\""
Oct 31 04:31:12.682552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.678884    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-6pgn5"]
Oct 31 04:31:12.772595 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.772558    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-metrics-certs\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.772801 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.772675    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96px\" (UniqueName: \"kubernetes.io/projected/90ef7101-8986-453d-aa53-e1a5c823b6cb-kube-api-access-s96px\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.773046 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.772710    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-webhook-cert\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.775401 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.775377    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-metrics-certs\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.775518 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.775427    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90ef7101-8986-453d-aa53-e1a5c823b6cb-webhook-cert\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.831603 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.831382    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96px\" (UniqueName: \"kubernetes.io/projected/90ef7101-8986-453d-aa53-e1a5c823b6cb-kube-api-access-s96px\") pod \"maas-controller-7c946769f6-6pgn5\" (UID: \"90ef7101-8986-453d-aa53-e1a5c823b6cb\") " pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.900623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.900520    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:12.989557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.986972    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:12.989557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:12.987035    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:13.007488 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:31:13.007288    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:31:13.034589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.034091    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:13.034589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.034153    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:13.355190 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.355162    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c946769f6-6pgn5"]
Oct 31 04:31:13.874484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.874438    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-6pgn5" event={"ID":"90ef7101-8986-453d-aa53-e1a5c823b6cb","Type":"ContainerStarted","Data":"178fdb17f52cf08b017b64e4c65bbc93721b199b538bfb0a37346725037e9522"}
Oct 31 04:31:13.997555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.997029    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:13.997555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:13.997096    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:14.031133 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.031094    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:14.031360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.031250    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:14.881686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.881631    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c946769f6-6pgn5" event={"ID":"90ef7101-8986-453d-aa53-e1a5c823b6cb","Type":"ContainerStarted","Data":"674dafab6f3a26b10ef8e465964f1825be0a174ff8ea12777c5dc2d4c335dc29"}
Oct 31 04:31:14.881897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.881711    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:14.985294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.985253    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:14.985465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.985335    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:14.988336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:14.988207    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7c946769f6-6pgn5" podStartSLOduration=2.102702739 podStartE2EDuration="2.988192371s" totalImagesPullingTime="885.489632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:14.98725457 +0000 UTC m=+5025.001816277" watchObservedRunningTime="2026-10-31 04:31:14.988192371 +0000 UTC m=+5025.002754074"
Oct 31 04:31:15.031650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:15.031613    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:15.032170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:15.031661    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:15.985915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:15.985871    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 31 04:31:15.986116 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:15.985961    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 31 04:31:16.031290 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:16.031249    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 31 04:31:16.031463 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:16.031320    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 31 04:31:16.989118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:16.989087    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:16.989622 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:16.989550    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:16.990166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:16.990144    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf"
Oct 31 04:31:17.035241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:17.035205    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:17.035586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:17.035568    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:17.036372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:17.036353    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g"
Oct 31 04:31:22.664841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.664803    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"]
Oct 31 04:31:22.668634 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.668612    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.714345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.714309    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.714696 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.714674    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226eca67-9a93-4f42-970d-cc145e14826c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.714867 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.714852    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758hv\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-kube-api-access-758hv\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715030 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715016    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226eca67-9a93-4f42-970d-cc145e14826c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715171    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715323    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715506    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715703    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.715875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.715861    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.773195 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.773161    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"]
Oct 31 04:31:22.817377 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817336    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817443    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817473    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817500    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817611    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817884 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817639    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226eca67-9a93-4f42-970d-cc145e14826c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.817954 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817909    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817943    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.817962    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758hv\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-kube-api-access-758hv\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.818031    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226eca67-9a93-4f42-970d-cc145e14826c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818139 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.818085    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.818157    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818371 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.818347    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226eca67-9a93-4f42-970d-cc145e14826c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.818441 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.818398    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.819889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.819869    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226eca67-9a93-4f42-970d-cc145e14826c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.820549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.820512    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226eca67-9a93-4f42-970d-cc145e14826c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.871060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.871030    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.871193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.871162    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758hv\" (UniqueName: \"kubernetes.io/projected/226eca67-9a93-4f42-970d-cc145e14826c-kube-api-access-758hv\") pod \"maas-default-gateway-openshift-default-6886b96f9-ddzvr\" (UID: \"226eca67-9a93-4f42-970d-cc145e14826c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:22.981914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:22.981874    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:23.400719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.400695    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"]
Oct 31 04:31:23.403323 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:31:23.403295    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226eca67_9a93_4f42_970d_cc145e14826c.slice/crio-a48c3319fb46cc64a469a5de6c8a5bdb0061be13f8d698827a4ea41dfaafdcd4 WatchSource:0}: Error finding container a48c3319fb46cc64a469a5de6c8a5bdb0061be13f8d698827a4ea41dfaafdcd4: Status 404 returned error can't find the container with id a48c3319fb46cc64a469a5de6c8a5bdb0061be13f8d698827a4ea41dfaafdcd4
Oct 31 04:31:23.405722 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.405691    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:23.405824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.405767    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:23.405824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.405798    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:31:23.940276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.940235    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" event={"ID":"226eca67-9a93-4f42-970d-cc145e14826c","Type":"ContainerStarted","Data":"94a5d8bf5d8b49f25764f182c4ec559c178b204a256003c1818bcac896bdb8f6"}
Oct 31 04:31:23.940698 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.940322    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" event={"ID":"226eca67-9a93-4f42-970d-cc145e14826c","Type":"ContainerStarted","Data":"a48c3319fb46cc64a469a5de6c8a5bdb0061be13f8d698827a4ea41dfaafdcd4"}
Oct 31 04:31:23.992557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:23.982461    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:24.050560 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:24.050467    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podStartSLOduration=2.050448208 podStartE2EDuration="2.050448208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:24.048883644 +0000 UTC m=+5034.063445351" watchObservedRunningTime="2026-10-31 04:31:24.050448208 +0000 UTC m=+5034.065009918"
Oct 31 04:31:24.984173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:24.984134    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:24.984830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:24.984798    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:25.896422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:25.896376    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c946769f6-6pgn5"
Oct 31 04:31:25.982797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:25.982742    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:25.983127 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:25.983084    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:27.004382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:27.000943    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:27.004382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:27.001009    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:27.983964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:27.983924    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:27.984311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:27.984281    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:29.003557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:28.999981    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:29.003557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:29.000044    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:29.982354 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:29.982319    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 31 04:31:29.982567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:29.982403    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 31 04:31:31.548016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:31.547975    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:31.548720 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:31.548698    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:31.549355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:31.549337    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr"
Oct 31 04:31:39.674334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:39.674304    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:31:42.979130 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:31:42.979091    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:31:45.630162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.630127    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"]
Oct 31 04:31:45.635310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.635290    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.686422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.686357    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"]
Oct 31 04:31:45.729641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.729606    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:31:45.729641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.729629    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:31:45.729894 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.729619    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:31:45.729937 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.729914    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:31:45.784028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.783995    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.784243 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.784043    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rf4\" (UniqueName: \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.784243 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.784180    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.885370 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.885270    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.885596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.885375    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.885596 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.885410    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rf4\" (UniqueName: \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.886068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.886046    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.887861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.887839    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.939254 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.939218    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rf4\" (UniqueName: \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\") pod \"maas-api-revoke-sub-f904bff9-b8wqq\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:45.949888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:45.949865    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:46.317799 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:46.317778    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"]
Oct 31 04:31:46.320217 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:31:46.320188    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e21a347_641e_4719_8e88_e1fad00c7ed7.slice/crio-353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72 WatchSource:0}: Error finding container 353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72: Status 404 returned error can't find the container with id 353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72
Oct 31 04:31:47.105981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:47.105940    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq" event={"ID":"5e21a347-641e-4719-8e88-e1fad00c7ed7","Type":"ContainerStarted","Data":"1f4a0654d12873bd18a3075f82dd6e4e2717405081900dbaf4fe04ff2f7cc573"}
Oct 31 04:31:47.106423 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:47.105997    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq" event={"ID":"5e21a347-641e-4719-8e88-e1fad00c7ed7","Type":"ContainerStarted","Data":"353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72"}
Oct 31 04:31:47.218900 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:47.218844    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq" podStartSLOduration=2.218829219 podStartE2EDuration="2.218829219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:47.217622348 +0000 UTC m=+5057.232184056" watchObservedRunningTime="2026-10-31 04:31:47.218829219 +0000 UTC m=+5057.233390927"
Oct 31 04:31:48.113497 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:48.113457    5831 generic.go:356] "Generic (PLEG): container finished" podID="5e21a347-641e-4719-8e88-e1fad00c7ed7" containerID="1f4a0654d12873bd18a3075f82dd6e4e2717405081900dbaf4fe04ff2f7cc573" exitCode=0
Oct 31 04:31:48.113929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:48.113553    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq" event={"ID":"5e21a347-641e-4719-8e88-e1fad00c7ed7","Type":"ContainerDied","Data":"1f4a0654d12873bd18a3075f82dd6e4e2717405081900dbaf4fe04ff2f7cc573"}
Oct 31 04:31:49.270313 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.270284    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:49.318733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.318701    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\") pod \"5e21a347-641e-4719-8e88-e1fad00c7ed7\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") "
Oct 31 04:31:49.318919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.318754    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\" (UniqueName: \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\") pod \"5e21a347-641e-4719-8e88-e1fad00c7ed7\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") "
Oct 31 04:31:49.318919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.318775    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\") pod \"5e21a347-641e-4719-8e88-e1fad00c7ed7\" (UID: \"5e21a347-641e-4719-8e88-e1fad00c7ed7\") "
Oct 31 04:31:49.319142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.319116    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca" pod "5e21a347-641e-4719-8e88-e1fad00c7ed7" (UID: "5e21a347-641e-4719-8e88-e1fad00c7ed7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:31:49.321100 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.321075    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca" pod "5e21a347-641e-4719-8e88-e1fad00c7ed7" (UID: "5e21a347-641e-4719-8e88-e1fad00c7ed7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:31:49.321207 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.321129    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4" pod "5e21a347-641e-4719-8e88-e1fad00c7ed7" (UID: "5e21a347-641e-4719-8e88-e1fad00c7ed7"). InnerVolumeSpecName "kube-api-access-78rf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:31:49.420260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.420178    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:49.420260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.420209    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rf4\" (UniqueName: \"kubernetes.io/projected/5e21a347-641e-4719-8e88-e1fad00c7ed7-kube-api-access-78rf4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:49.420260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:49.420219    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e21a347-641e-4719-8e88-e1fad00c7ed7-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:31:50.133503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:50.133465    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"
Oct 31 04:31:50.135505 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:50.135477    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq" event={"ID":"5e21a347-641e-4719-8e88-e1fad00c7ed7","Type":"ContainerDied","Data":"353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72"}
Oct 31 04:31:50.135620 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:50.135549    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353995a96ee171cccbe7c823510aa655ae83cfd10f8da86271188b05575cba72"
Oct 31 04:31:58.468566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:31:58.468509    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:32:06.686966 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:06.686930    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:32:07.693198 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.693160    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"]
Oct 31 04:32:07.693688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.693651    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5e21a347-641e-4719-8e88-e1fad00c7ed7" containerName="revoke-keys"
Oct 31 04:32:07.693688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.693664    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e21a347-641e-4719-8e88-e1fad00c7ed7" containerName="revoke-keys"
Oct 31 04:32:07.693803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.693765    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e21a347-641e-4719-8e88-e1fad00c7ed7" containerName="revoke-keys"
Oct 31 04:32:07.697103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.697083    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.794637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.794604    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"]
Oct 31 04:32:07.878788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.878750    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.878977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.878871    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjsx\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-kube-api-access-zdjsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.878977 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.878943    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.878985    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.879031    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.879083    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.879132    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879231 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.879189    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.879231 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.879216    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980483    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980586    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980609 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980607    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980626    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980671    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980703    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980733    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980762    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.980938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980847    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjsx\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-kube-api-access-zdjsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.981248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.980990    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.981248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.981039    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.981248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.981067    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.981417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.981344    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.981492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.981468    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.983613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.983583    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:07.983753 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:07.983634    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:08.034703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.034674    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:08.034893 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.034872    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjsx\" (UniqueName: \"kubernetes.io/projected/e81cdb3c-2ad9-4df5-af12-8d20de42ce47-kube-api-access-zdjsx\") pod \"maas-default-gateway-openshift-default-6886b96f9-lqmhm\" (UID: \"e81cdb3c-2ad9-4df5-af12-8d20de42ce47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:08.314339 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.314239    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:08.739870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.739840    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"]
Oct 31 04:32:08.743105 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:32:08.743070    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81cdb3c_2ad9_4df5_af12_8d20de42ce47.slice/crio-d0498e79fa0d529f760a40128684e5fc29047458d0fd5ef571a1fe547924930f WatchSource:0}: Error finding container d0498e79fa0d529f760a40128684e5fc29047458d0fd5ef571a1fe547924930f: Status 404 returned error can't find the container with id d0498e79fa0d529f760a40128684e5fc29047458d0fd5ef571a1fe547924930f
Oct 31 04:32:08.745778 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.745733    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:32:08.745878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.745828    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:32:08.745878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:08.745858    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:32:09.310863 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.310819    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" event={"ID":"e81cdb3c-2ad9-4df5-af12-8d20de42ce47","Type":"ContainerStarted","Data":"5138be8733fef8d9c16b379df010cbd580bc6b1c4355b54159aca4e4c9ad48bd"}
Oct 31 04:32:09.311057 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.310884    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" event={"ID":"e81cdb3c-2ad9-4df5-af12-8d20de42ce47","Type":"ContainerStarted","Data":"d0498e79fa0d529f760a40128684e5fc29047458d0fd5ef571a1fe547924930f"}
Oct 31 04:32:09.315262 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.315244    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:09.316955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.316931    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:09.317062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.316973    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:09.419316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:09.419263    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podStartSLOduration=2.419248665 podStartE2EDuration="2.419248665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:09.417279809 +0000 UTC m=+5079.431841517" watchObservedRunningTime="2026-10-31 04:32:09.419248665 +0000 UTC m=+5079.433810372"
Oct 31 04:32:10.314773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:10.314739    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:10.315208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:10.314792    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:11.315641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:11.315603    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:11.316049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:11.315670    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:12.316088 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:12.316051    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:12.316649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:12.316116    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:13.091080 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:32:13.091035    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:32:13.317994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:13.317953    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:13.318619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:13.318583    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:14.315867 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:14.315831    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:14.316197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:14.316167    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:15.314777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:15.314737    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:15.315224 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:15.314797    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:16.315248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:16.315209    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body=
Oct 31 04:32:16.315810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:16.315274    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused"
Oct 31 04:32:16.395405 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:16.395372    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:32:18.181191 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.181149    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"]
Oct 31 04:32:18.185485 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.185460    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.234467 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.234434    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:32:18.260499 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.260439    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.272786 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.272750    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzjz\" (UniqueName: \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.272991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.272845    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.280423 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.280387    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:32:18.280635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.280486    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:32:18.280801 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.280772    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:32:18.283239 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.283212    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"]
Oct 31 04:32:18.356703 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.356659    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:32:18.356912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.356723    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:32:18.373781 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.373739    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzjz\" (UniqueName: \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.373956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.373817    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.373956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.373856    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.374542 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.374500    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.376475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.376453    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.427004 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.426963    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzjz\" (UniqueName: \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\") pod \"maas-api-revoke-sub-f03629db-fpt4m\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.499230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.499136    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:18.892618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:18.892584    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"]
Oct 31 04:32:18.894679 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:32:18.894648    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5c4c7a_50ad_46e1_95e5_551de28e5372.slice/crio-78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd WatchSource:0}: Error finding container 78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd: Status 404 returned error can't find the container with id 78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd
Oct 31 04:32:19.398306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.397789    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:32:19.398306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.397864    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:32:19.413641 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.413601    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m" event={"ID":"1a5c4c7a-50ad-46e1-95e5-551de28e5372","Type":"ContainerStarted","Data":"20d658592d8c5e6be9b4e99fda18c2851040685ec182a16f0d25758f4dbdc49d"}
Oct 31 04:32:19.413860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.413675    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m" event={"ID":"1a5c4c7a-50ad-46e1-95e5-551de28e5372","Type":"ContainerStarted","Data":"78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd"}
Oct 31 04:32:19.507602 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.507491    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:19.508134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.508106    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:19.508709 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.508693    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm"
Oct 31 04:32:19.520982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:19.520939    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m" podStartSLOduration=1.5209257059999999 podStartE2EDuration="1.520925706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:19.517710081 +0000 UTC m=+5089.532271789" watchObservedRunningTime="2026-10-31 04:32:19.520925706 +0000 UTC m=+5089.535487414"
Oct 31 04:32:20.423240 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:20.423198    5831 generic.go:356] "Generic (PLEG): container finished" podID="1a5c4c7a-50ad-46e1-95e5-551de28e5372" containerID="20d658592d8c5e6be9b4e99fda18c2851040685ec182a16f0d25758f4dbdc49d" exitCode=0
Oct 31 04:32:20.423784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:20.423298    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m" event={"ID":"1a5c4c7a-50ad-46e1-95e5-551de28e5372","Type":"ContainerDied","Data":"20d658592d8c5e6be9b4e99fda18c2851040685ec182a16f0d25758f4dbdc49d"}
Oct 31 04:32:21.570543 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.570503    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:21.620920 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.620887    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\") pod \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") "
Oct 31 04:32:21.620920 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.620926    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\" (UniqueName: \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\") pod \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") "
Oct 31 04:32:21.621141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.620981    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\") pod \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\" (UID: \"1a5c4c7a-50ad-46e1-95e5-551de28e5372\") "
Oct 31 04:32:21.621311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.621282    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca" pod "1a5c4c7a-50ad-46e1-95e5-551de28e5372" (UID: "1a5c4c7a-50ad-46e1-95e5-551de28e5372"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:21.623307 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.623280    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca" pod "1a5c4c7a-50ad-46e1-95e5-551de28e5372" (UID: "1a5c4c7a-50ad-46e1-95e5-551de28e5372"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:21.623430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.623373    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz" pod "1a5c4c7a-50ad-46e1-95e5-551de28e5372" (UID: "1a5c4c7a-50ad-46e1-95e5-551de28e5372"). InnerVolumeSpecName "kube-api-access-hbzjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:21.722957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.722850    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:21.722957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.722891    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzjz\" (UniqueName: \"kubernetes.io/projected/1a5c4c7a-50ad-46e1-95e5-551de28e5372-kube-api-access-hbzjz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:21.722957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:21.722907    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5c4c7a-50ad-46e1-95e5-551de28e5372-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:22.436371 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:22.436341    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"
Oct 31 04:32:22.436585 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:22.436379    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m" event={"ID":"1a5c4c7a-50ad-46e1-95e5-551de28e5372","Type":"ContainerDied","Data":"78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd"}
Oct 31 04:32:22.436585 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:22.436425    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b326fea97271dcf6652c9cc461e3cbc128924941b3ebaff1ebdfb58fc612dd"
Oct 31 04:32:25.088081 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:25.088044    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:32:26.042679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:26.042638    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"]
Oct 31 04:32:26.045500 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:26.045465    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a715685-l2rbr"]
Oct 31 04:32:26.134923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:26.134891    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0c2efa-08be-43b8-a55e-e371e8a3ed16" path="/var/lib/kubelet/pods/2d0c2efa-08be-43b8-a55e-e371e8a3ed16/volumes"
Oct 31 04:32:38.171446 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.171409    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"]
Oct 31 04:32:38.172132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.171864    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1a5c4c7a-50ad-46e1-95e5-551de28e5372" containerName="revoke-keys"
Oct 31 04:32:38.172132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.171876    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5c4c7a-50ad-46e1-95e5-551de28e5372" containerName="revoke-keys"
Oct 31 04:32:38.172132 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.171942    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5c4c7a-50ad-46e1-95e5-551de28e5372" containerName="revoke-keys"
Oct 31 04:32:38.174947 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.174927    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.257052 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.257017    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.257265 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.257098    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.257347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.257261    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5wf\" (UniqueName: \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.270246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.270215    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:32:38.270456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.270227    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:32:38.270456 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.270227    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:32:38.270650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.270288    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:32:38.272851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.272830    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"]
Oct 31 04:32:38.358341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.358308    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.358584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.358393    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5wf\" (UniqueName: \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.358584 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.358424    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.359271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.359247    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.360978 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.360956    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.419411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.419373    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5wf\" (UniqueName: \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\") pod \"maas-api-revoke-sub-0165d7d2-59phs\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.488697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.488660    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:38.768302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.768230    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:32:38.776241 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.776218    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:32:38.777836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.777815    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:32:38.781930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.781911    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:32:38.787463 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.787445    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:32:38.790939 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.790924    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:32:38.863348 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:38.863321    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"]
Oct 31 04:32:38.866311 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:32:38.866277    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dfca28_624c_498c_b20b_1296054918d3.slice/crio-cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61 WatchSource:0}: Error finding container cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61: Status 404 returned error can't find the container with id cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61
Oct 31 04:32:39.588390 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:39.588348    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs" event={"ID":"37dfca28-624c-498c-b20b-1296054918d3","Type":"ContainerStarted","Data":"a661c45a6f18fc67014f3d20e264ef098c361041c188f85fa443ee2ceb440854"}
Oct 31 04:32:39.588908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:39.588414    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs" event={"ID":"37dfca28-624c-498c-b20b-1296054918d3","Type":"ContainerStarted","Data":"cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61"}
Oct 31 04:32:39.695454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:39.695404    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs" podStartSLOduration=1.695390127 podStartE2EDuration="1.695390127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:39.693618789 +0000 UTC m=+5109.708180498" watchObservedRunningTime="2026-10-31 04:32:39.695390127 +0000 UTC m=+5109.709951829"
Oct 31 04:32:40.595629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:40.595594    5831 generic.go:356] "Generic (PLEG): container finished" podID="37dfca28-624c-498c-b20b-1296054918d3" containerID="a661c45a6f18fc67014f3d20e264ef098c361041c188f85fa443ee2ceb440854" exitCode=0
Oct 31 04:32:40.596118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:40.595683    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs" event={"ID":"37dfca28-624c-498c-b20b-1296054918d3","Type":"ContainerDied","Data":"a661c45a6f18fc67014f3d20e264ef098c361041c188f85fa443ee2ceb440854"}
Oct 31 04:32:41.737846 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.737815    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:41.791303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.791267    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\" (UniqueName: \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\") pod \"37dfca28-624c-498c-b20b-1296054918d3\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") "
Oct 31 04:32:41.791501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.791315    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\") pod \"37dfca28-624c-498c-b20b-1296054918d3\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") "
Oct 31 04:32:41.791501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.791419    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\") pod \"37dfca28-624c-498c-b20b-1296054918d3\" (UID: \"37dfca28-624c-498c-b20b-1296054918d3\") "
Oct 31 04:32:41.791788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.791764    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca" pod "37dfca28-624c-498c-b20b-1296054918d3" (UID: "37dfca28-624c-498c-b20b-1296054918d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:41.793586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.793563    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca" pod "37dfca28-624c-498c-b20b-1296054918d3" (UID: "37dfca28-624c-498c-b20b-1296054918d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:41.793694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.793673    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf" pod "37dfca28-624c-498c-b20b-1296054918d3" (UID: "37dfca28-624c-498c-b20b-1296054918d3"). InnerVolumeSpecName "kube-api-access-5d5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:41.892854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.892771    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37dfca28-624c-498c-b20b-1296054918d3-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:41.892854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.892801    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5wf\" (UniqueName: \"kubernetes.io/projected/37dfca28-624c-498c-b20b-1296054918d3-kube-api-access-5d5wf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:41.892854 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:41.892816    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37dfca28-624c-498c-b20b-1296054918d3-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:42.609121 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:42.609091    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"
Oct 31 04:32:42.609340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:42.609126    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs" event={"ID":"37dfca28-624c-498c-b20b-1296054918d3","Type":"ContainerDied","Data":"cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61"}
Oct 31 04:32:42.609340 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:42.609177    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cceda13630ffb46ebf4d395bafa550ddec2a25a7007cba0c89bf7384c9366a61"
Oct 31 04:32:42.977417 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:32:42.977374    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:32:49.889448 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.889411    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"]
Oct 31 04:32:49.889923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.889907    5831 cpu_manager.go:414] "containerMap: removing container" podUID="37dfca28-624c-498c-b20b-1296054918d3" containerName="revoke-keys"
Oct 31 04:32:49.889986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.889924    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37dfca28-624c-498c-b20b-1296054918d3" containerName="revoke-keys"
Oct 31 04:32:49.890045 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.890012    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="37dfca28-624c-498c-b20b-1296054918d3" containerName="revoke-keys"
Oct 31 04:32:49.893719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.893698    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:49.946734 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.946696    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"]
Oct 31 04:32:49.969949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.969920    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:49.970159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.970051    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:49.970159 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.970105    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:49.988668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.988641    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:32:49.988668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.988649    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:32:49.988874 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.988725    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:32:49.988874 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:49.988785    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:32:50.071421 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.071391    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.071628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.071474    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.071628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.071554    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.072226 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.072202    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.074071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.074050    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.124288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.124258    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\") pod \"maas-api-revoke-sub-6b19338f-trl4w\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.207311 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.207284    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:50.578619 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.578592    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"]
Oct 31 04:32:50.581236 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:32:50.581197    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f562c79_c069_44c2_b7fd_736d7a28d786.slice/crio-d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d WatchSource:0}: Error finding container d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d: Status 404 returned error can't find the container with id d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d
Oct 31 04:32:50.657659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:50.657624    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w" event={"ID":"7f562c79-c069-44c2-b7fd-736d7a28d786","Type":"ContainerStarted","Data":"d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d"}
Oct 31 04:32:51.666519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:51.666485    5831 generic.go:356] "Generic (PLEG): container finished" podID="7f562c79-c069-44c2-b7fd-736d7a28d786" containerID="5353139b718a3d483aa3582250bf09150a55634a3d080ea56e3a5f9861713322" exitCode=0
Oct 31 04:32:51.666955 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:51.666575    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w" event={"ID":"7f562c79-c069-44c2-b7fd-736d7a28d786","Type":"ContainerDied","Data":"5353139b718a3d483aa3582250bf09150a55634a3d080ea56e3a5f9861713322"}
Oct 31 04:32:52.813491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.813467    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:52.910835 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.910806    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\") pod \"7f562c79-c069-44c2-b7fd-736d7a28d786\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") "
Oct 31 04:32:52.911033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.910937    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\") pod \"7f562c79-c069-44c2-b7fd-736d7a28d786\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") "
Oct 31 04:32:52.911033 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.910956    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\") pod \"7f562c79-c069-44c2-b7fd-736d7a28d786\" (UID: \"7f562c79-c069-44c2-b7fd-736d7a28d786\") "
Oct 31 04:32:52.911242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.911214    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca" pod "7f562c79-c069-44c2-b7fd-736d7a28d786" (UID: "7f562c79-c069-44c2-b7fd-736d7a28d786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:52.913206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.913177    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h" pod "7f562c79-c069-44c2-b7fd-736d7a28d786" (UID: "7f562c79-c069-44c2-b7fd-736d7a28d786"). InnerVolumeSpecName "kube-api-access-6q79h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:52.913342 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:52.913261    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca" pod "7f562c79-c069-44c2-b7fd-736d7a28d786" (UID: "7f562c79-c069-44c2-b7fd-736d7a28d786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:53.012302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.012267    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/7f562c79-c069-44c2-b7fd-736d7a28d786-kube-api-access-6q79h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:53.012302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.012300    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:53.012302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.012312    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f562c79-c069-44c2-b7fd-736d7a28d786-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:32:53.680711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.680679    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"
Oct 31 04:32:53.680910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.680710    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w" event={"ID":"7f562c79-c069-44c2-b7fd-736d7a28d786","Type":"ContainerDied","Data":"d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d"}
Oct 31 04:32:53.680910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:32:53.680770    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d788e470f410524474a9920c559ca38cd94e5a96773733e3ad4698f2fdc9988d"
Oct 31 04:33:02.868252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:02.868172    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:33:07.474735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.474703    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:33:07.475134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.475117    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7f562c79-c069-44c2-b7fd-736d7a28d786" containerName="revoke-keys"
Oct 31 04:33:07.475134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.475128    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f562c79-c069-44c2-b7fd-736d7a28d786" containerName="revoke-keys"
Oct 31 04:33:07.475246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.475221    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f562c79-c069-44c2-b7fd-736d7a28d786" containerName="revoke-keys"
Oct 31 04:33:07.478873 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.478845    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.489131 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.489103    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.489303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.489276    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqf9\" (UniqueName: \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.527917 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.527869    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7010fecc-kserve-self-signed-certs\""
Oct 31 04:33:07.573765 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.573721    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7010fecc-epp-sa-dockercfg-4clmj\""
Oct 31 04:33:07.576369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.576345    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:33:07.590288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.590252    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqf9\" (UniqueName: \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.590446 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.590364    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.593305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.593269    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.650623 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.650575    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqf9\" (UniqueName: \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\") pod \"e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") " pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:07.796107 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:07.796015    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:08.170086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:08.170054    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:33:08.172054 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:33:08.172017    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91b8ea2_89b8_45cf_b71e_bad3ac31bb1e.slice/crio-d0708e673dbbe5af02b94acb00588b213b2623bd2578aebc55e778e3221b26cc WatchSource:0}: Error finding container d0708e673dbbe5af02b94acb00588b213b2623bd2578aebc55e778e3221b26cc: Status 404 returned error can't find the container with id d0708e673dbbe5af02b94acb00588b213b2623bd2578aebc55e778e3221b26cc
Oct 31 04:33:08.808635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:08.808598    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" event={"ID":"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e","Type":"ContainerStarted","Data":"d0708e673dbbe5af02b94acb00588b213b2623bd2578aebc55e778e3221b26cc"}
Oct 31 04:33:10.835889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:10.835852    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" event={"ID":"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e","Type":"ContainerStarted","Data":"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"}
Oct 31 04:33:10.836333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:10.835945    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:10.946317 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:10.946260    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podStartSLOduration=1.806009854 podStartE2EDuration="3.9462451s" totalImagesPullingTime="2.140235246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:10.944409342 +0000 UTC m=+5140.958971074" watchObservedRunningTime="2026-10-31 04:33:10.9462451 +0000 UTC m=+5140.960806808"
Oct 31 04:33:12.977905 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:33:12.977870    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:33:14.454028 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:14.453992    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:33:17.599682 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:17.599642    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:33:17.600324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:17.600287    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:33:18.088664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:18.088625    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:33:18.088987 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:18.088956    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:33:20.044765 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:20.044730    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"]
Oct 31 04:33:20.091233 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:20.091191    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a390f2-fwjn9"]
Oct 31 04:33:20.135950 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:20.135915    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5fe49c-d378-4dba-a5d1-1f1d0be5df03" path="/var/lib/kubelet/pods/de5fe49c-d378-4dba-a5d1-1f1d0be5df03/volumes"
Oct 31 04:33:28.041909 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:28.041862    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"]
Oct 31 04:33:28.044386 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:28.044359    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c7eca-qdcps"]
Oct 31 04:33:28.133762 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:28.133724    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc49af2-1de3-45b6-a5ea-de0f7ab98b08" path="/var/lib/kubelet/pods/0dc49af2-1de3-45b6-a5ea-de0f7ab98b08/volumes"
Oct 31 04:33:32.041667 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:32.041614    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"]
Oct 31 04:33:32.087613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:32.087579    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2eb7e3-8g6xl"]
Oct 31 04:33:32.133662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:32.133631    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9687d6-a017-4c81-a444-e30cffd75bb0" path="/var/lib/kubelet/pods/2c9687d6-a017-4c81-a444-e30cffd75bb0/volumes"
Oct 31 04:33:37.470906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:37.470868    5831 scope.go:122] "RemoveContainer" containerID="1c901d8ba3b2f0fab409f51d6b32c50c4ae4a0153af682acf196eebf0a6fb4b0"
Oct 31 04:33:41.043618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:41.043579    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"]
Oct 31 04:33:41.089681 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:41.089647    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a7e02b-6g58h"]
Oct 31 04:33:41.845276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:41.845242    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:33:42.134621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:42.134507    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56176d7-3cb7-4685-89cb-1192a32124a0" path="/var/lib/kubelet/pods/d56176d7-3cb7-4685-89cb-1192a32124a0/volumes"
Oct 31 04:33:42.978478 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:33:42.978438    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:33:53.705155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:33:53.705118    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:34:12.977453 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:34:12.977415    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:34:18.060592 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:18.060554    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:34:18.077763 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:18.077718    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:34:21.845252 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:21.845213    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:34:29.690511 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:29.690418    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:34:31.845063 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:31.845021    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:34:33.092740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:33.092703    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" containerID="cri-o://f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd" gracePeriod=45
Oct 31 04:34:37.496164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:37.496131    5831 scope.go:122] "RemoveContainer" containerID="a92e69ddb3abffb9b4ecb79d35b9b72e68b603582fd0b56ee9cff5430e12bdc6"
Oct 31 04:34:37.505816 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:37.505796    5831 scope.go:122] "RemoveContainer" containerID="42485c7dd6acd0bd5c0c48c49e902f1bc701983112fdd1df8705314fb340ecd6"
Oct 31 04:34:37.515244 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:37.515217    5831 scope.go:122] "RemoveContainer" containerID="e2d5c7b051e536efc82895ca5d079e98749bd64d4d7032656232666b866b8aec"
Oct 31 04:34:37.524862 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:37.524840    5831 scope.go:122] "RemoveContainer" containerID="8aa837b57ff67d18a6b3895a5bce9999f034ba97bd8567b0b45fd2c1115cce14"
Oct 31 04:34:41.864558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:41.861644    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:34:41.864558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:41.861800    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:34:42.979036 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:34:42.978999    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:34:51.844472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:51.844434    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:34:55.276180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:34:55.276144    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:35:01.844845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:01.844804    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:35:03.351484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.351421    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:35:03.473543 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.473493    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\") pod \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") "
Oct 31 04:35:03.473740 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.473591    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\" (UniqueName: \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\") pod \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\" (UID: \"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e\") "
Oct 31 04:35:03.476007 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.475974    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9" pod "c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" (UID: "c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e"). InnerVolumeSpecName "kube-api-access-ztqf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:35:03.476134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.476053    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs" pod "c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" (UID: "c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:35:03.575184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.575155    5831 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-tls-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:35:03.575184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.575183    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqf9\" (UniqueName: \"kubernetes.io/projected/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e-kube-api-access-ztqf9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:35:03.865593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.865557    5831 generic.go:356] "Generic (PLEG): container finished" podID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerID="f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd" exitCode=0
Oct 31 04:35:03.865788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.865606    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"
Oct 31 04:35:03.865788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.865641    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" event={"ID":"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e","Type":"ContainerDied","Data":"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"}
Oct 31 04:35:03.865788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.865700    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x" event={"ID":"c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e","Type":"ContainerDied","Data":"d0708e673dbbe5af02b94acb00588b213b2623bd2578aebc55e778e3221b26cc"}
Oct 31 04:35:03.865788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.865724    5831 scope.go:122] "RemoveContainer" containerID="f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"
Oct 31 04:35:03.876015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.875994    5831 scope.go:122] "RemoveContainer" containerID="f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"
Oct 31 04:35:03.876285 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:35:03.876266    5831 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd\": container with ID starting with f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd not found: ID does not exist" containerID="f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"
Oct 31 04:35:03.876342 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:03.876295    5831 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd"} err="failed to get container status \"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd\": rpc error: code = NotFound desc = could not find container \"f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd\": container with ID starting with f89b922c4d07b14cf62de26a7897cc11b7f9ebd6c9d44d9a59f908b596c8a5dd not found: ID does not exist"
Oct 31 04:35:04.016578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:04.016546    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:35:04.062395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:04.062360    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-router-scheduler-5b498b48f7-bkf6x"]
Oct 31 04:35:04.134597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:04.134493    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" path="/var/lib/kubelet/pods/c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e/volumes"
Oct 31 04:35:12.978516 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:35:12.978473    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:35:14.164060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:14.164021    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:35:39.557160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:39.557125    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:35:42.978924 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:35:42.978883    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:35:54.421062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.421027    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"]
Oct 31 04:35:54.421604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.421447    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main"
Oct 31 04:35:54.421604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.421457    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main"
Oct 31 04:35:54.421604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.421585    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91b8ea2-89b8-45cf-b71e-bad3ac31bb1e" containerName="main"
Oct 31 04:35:54.426120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.426097    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.473152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.473115    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.473316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.473158    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jpb\" (UniqueName: \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.473316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.473231    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.521000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.520965    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:35:54.521000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.520965    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:35:54.521000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.520965    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:35:54.521000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.520996    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:35:54.523572 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.523515    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"]
Oct 31 04:35:54.574120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.574087    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.574120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.574122    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jpb\" (UniqueName: \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.574358 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.574175    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.574738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.574714    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.576830 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.576808    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.627184 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.627152    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jpb\" (UniqueName: \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\") pod \"maas-api-revoke-sub-5bb9ef3d-r2jsz\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:54.739791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:54.739756    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:55.118079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:55.118053    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"]
Oct 31 04:35:55.120573 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:35:55.120523    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974f3698_2b43_4f5f_ab8c_b57935de1d52.slice/crio-f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8 WatchSource:0}: Error finding container f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8: Status 404 returned error can't find the container with id f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8
Oct 31 04:35:55.182247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:55.182211    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz" event={"ID":"974f3698-2b43-4f5f-ab8c-b57935de1d52","Type":"ContainerStarted","Data":"f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8"}
Oct 31 04:35:56.192091 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:56.192052    5831 generic.go:356] "Generic (PLEG): container finished" podID="974f3698-2b43-4f5f-ab8c-b57935de1d52" containerID="68c62049acfa514c79ee2836dd80dc2d464f8299cecfc363dafabc52b94b669a" exitCode=0
Oct 31 04:35:56.192628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:56.192144    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz" event={"ID":"974f3698-2b43-4f5f-ab8c-b57935de1d52","Type":"ContainerDied","Data":"68c62049acfa514c79ee2836dd80dc2d464f8299cecfc363dafabc52b94b669a"}
Oct 31 04:35:57.366869 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.366839    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:35:57.505986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.505887    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\" (UniqueName: \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\") pod \"974f3698-2b43-4f5f-ab8c-b57935de1d52\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") "
Oct 31 04:35:57.505986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.505951    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\") pod \"974f3698-2b43-4f5f-ab8c-b57935de1d52\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") "
Oct 31 04:35:57.505986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.505978    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\") pod \"974f3698-2b43-4f5f-ab8c-b57935de1d52\" (UID: \"974f3698-2b43-4f5f-ab8c-b57935de1d52\") "
Oct 31 04:35:57.506739 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.506705    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca" pod "974f3698-2b43-4f5f-ab8c-b57935de1d52" (UID: "974f3698-2b43-4f5f-ab8c-b57935de1d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:35:57.508440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.508410    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca" pod "974f3698-2b43-4f5f-ab8c-b57935de1d52" (UID: "974f3698-2b43-4f5f-ab8c-b57935de1d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:35:57.508566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.508514    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb" pod "974f3698-2b43-4f5f-ab8c-b57935de1d52" (UID: "974f3698-2b43-4f5f-ab8c-b57935de1d52"). InnerVolumeSpecName "kube-api-access-h5jpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:35:57.607038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.607002    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jpb\" (UniqueName: \"kubernetes.io/projected/974f3698-2b43-4f5f-ab8c-b57935de1d52-kube-api-access-h5jpb\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:35:57.607038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.607033    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:35:57.607038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:57.607044    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974f3698-2b43-4f5f-ab8c-b57935de1d52-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:35:58.208864 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:58.208786    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz" event={"ID":"974f3698-2b43-4f5f-ab8c-b57935de1d52","Type":"ContainerDied","Data":"f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8"}
Oct 31 04:35:58.208864 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:58.208857    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f518163ce1480712ba1a48004e7810de4dfa1207a2d6e8322eec8369b5fe49d8"
Oct 31 04:35:58.209079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:35:58.208795    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"
Oct 31 04:36:00.042856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:00.042824    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"]
Oct 31 04:36:00.088343 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:00.088311    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cwjv4"]
Oct 31 04:36:00.134359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:00.134326    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b86c892-984f-41c4-bc7c-30c0cee1127b" path="/var/lib/kubelet/pods/6b86c892-984f-41c4-bc7c-30c0cee1127b/volumes"
Oct 31 04:36:07.393320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:07.393276    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:36:12.978392 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:36:12.978357    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:36:21.349882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:21.349849    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:36:37.587541 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:37.587507    5831 scope.go:122] "RemoveContainer" containerID="fd6e968cffe3dbf8bba07a0eb9fd46c7b61783f6c7d4dd894c271fbffbcb7130"
Oct 31 04:36:42.977666 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:36:42.977629    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:36:50.043546 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:50.043499    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"]
Oct 31 04:36:50.089644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:50.089609    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b8wqq"]
Oct 31 04:36:50.134473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:36:50.134436    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e21a347-641e-4719-8e88-e1fad00c7ed7" path="/var/lib/kubelet/pods/5e21a347-641e-4719-8e88-e1fad00c7ed7/volumes"
Oct 31 04:37:10.585438 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:10.585401    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:37:13.087153 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:37:13.087110    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:37:22.043647 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:22.043608    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"]
Oct 31 04:37:22.089581 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:22.089515    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fpt4m"]
Oct 31 04:37:22.133692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:22.133658    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5c4c7a-50ad-46e1-95e5-551de28e5372" path="/var/lib/kubelet/pods/1a5c4c7a-50ad-46e1-95e5-551de28e5372/volumes"
Oct 31 04:37:33.269517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:33.269390    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:37:37.611994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:37.611963    5831 scope.go:122] "RemoveContainer" containerID="8656bc0ac24a5c098b30a8372469b748935efd5d9c3d534b1cdcdc97354655a3"
Oct 31 04:37:38.818267 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.818236    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:37:38.826283 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.826257    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:37:38.829327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.829307    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:37:38.842594 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.842562    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:37:38.845904 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.845884    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:37:38.849326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:38.849310    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:37:42.044960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:42.044924    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"]
Oct 31 04:37:42.091245 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:42.091207    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-59phs"]
Oct 31 04:37:42.134488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:42.134453    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37dfca28-624c-498c-b20b-1296054918d3" path="/var/lib/kubelet/pods/37dfca28-624c-498c-b20b-1296054918d3/volumes"
Oct 31 04:37:42.977626 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:37:42.977588    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:37:53.043247 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:53.043207    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"]
Oct 31 04:37:53.089403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:53.089368    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-trl4w"]
Oct 31 04:37:54.134369 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:54.134335    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f562c79-c069-44c2-b7fd-736d7a28d786" path="/var/lib/kubelet/pods/7f562c79-c069-44c2-b7fd-736d7a28d786/volumes"
Oct 31 04:37:59.220697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.220664    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"]
Oct 31 04:37:59.221092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.221069    5831 cpu_manager.go:414] "containerMap: removing container" podUID="974f3698-2b43-4f5f-ab8c-b57935de1d52" containerName="revoke-keys"
Oct 31 04:37:59.221092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.221079    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974f3698-2b43-4f5f-ab8c-b57935de1d52" containerName="revoke-keys"
Oct 31 04:37:59.221173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.221163    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="974f3698-2b43-4f5f-ab8c-b57935de1d52" containerName="revoke-keys"
Oct 31 04:37:59.225757 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.225731    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.311742 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.311703    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.311923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.311771    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qv4n\" (UniqueName: \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.311923 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.311860    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.321291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.321254    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:37:59.321291 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.321260    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:37:59.321502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.321331    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:37:59.321502 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.321378    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:37:59.326287 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.326243    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"]
Oct 31 04:37:59.412948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.412917    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.413151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.412966    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qv4n\" (UniqueName: \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.413151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.413003    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.413724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.413693    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.415483 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.415464    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.466490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.466459    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qv4n\" (UniqueName: \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\") pod \"maas-api-revoke-sub-452b6d7b-4p85w\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.538729 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.538648    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:37:59.910493 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:37:59.910395    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"]
Oct 31 04:37:59.917734 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:37:59.917701    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cdacda_3334_40a5_b11e_1ba6f9fb7231.slice/crio-4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13 WatchSource:0}: Error finding container 4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13: Status 404 returned error can't find the container with id 4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13
Oct 31 04:38:00.121588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:00.121551    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w" event={"ID":"29cdacda-3334-40a5-b11e-1ba6f9fb7231","Type":"ContainerStarted","Data":"4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13"}
Oct 31 04:38:00.663508 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:00.663426    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:38:01.129427 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:01.129393    5831 generic.go:356] "Generic (PLEG): container finished" podID="29cdacda-3334-40a5-b11e-1ba6f9fb7231" containerID="af80627d51d365a95be91fd4375de9b72c2b49e1cc175340ffa4b4dc182293d1" exitCode=0
Oct 31 04:38:01.129648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:01.129424    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w" event={"ID":"29cdacda-3334-40a5-b11e-1ba6f9fb7231","Type":"ContainerDied","Data":"af80627d51d365a95be91fd4375de9b72c2b49e1cc175340ffa4b4dc182293d1"}
Oct 31 04:38:02.434649 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.434241    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:38:02.448687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.447914    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\") pod \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") "
Oct 31 04:38:02.448687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.447970    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\" (UniqueName: \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\") pod \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") "
Oct 31 04:38:02.448687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.448090    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\") pod \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\" (UID: \"29cdacda-3334-40a5-b11e-1ba6f9fb7231\") "
Oct 31 04:38:02.449945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.449888    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca" pod "29cdacda-3334-40a5-b11e-1ba6f9fb7231" (UID: "29cdacda-3334-40a5-b11e-1ba6f9fb7231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:02.467589 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.467485    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n" pod "29cdacda-3334-40a5-b11e-1ba6f9fb7231" (UID: "29cdacda-3334-40a5-b11e-1ba6f9fb7231"). InnerVolumeSpecName "kube-api-access-9qv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:02.468994 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.468950    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca" pod "29cdacda-3334-40a5-b11e-1ba6f9fb7231" (UID: "29cdacda-3334-40a5-b11e-1ba6f9fb7231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:38:02.549521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.549227    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:02.549521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.549268    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29cdacda-3334-40a5-b11e-1ba6f9fb7231-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:02.549521 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:02.549287    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qv4n\" (UniqueName: \"kubernetes.io/projected/29cdacda-3334-40a5-b11e-1ba6f9fb7231-kube-api-access-9qv4n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:03.173454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:03.173417    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w" event={"ID":"29cdacda-3334-40a5-b11e-1ba6f9fb7231","Type":"ContainerDied","Data":"4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13"}
Oct 31 04:38:03.173789 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:03.173765    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efcfff5564bcbd7df2a8010afa726a5bc07219b87b75099dde1ce2e14418a13"
Oct 31 04:38:03.174578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:03.174553    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"
Oct 31 04:38:12.298365 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:12.298329    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:38:12.981522 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:38:12.981479    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:38:18.725445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.723724    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"]
Oct 31 04:38:18.725445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.724249    5831 cpu_manager.go:414] "containerMap: removing container" podUID="29cdacda-3334-40a5-b11e-1ba6f9fb7231" containerName="revoke-keys"
Oct 31 04:38:18.725445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.724264    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cdacda-3334-40a5-b11e-1ba6f9fb7231" containerName="revoke-keys"
Oct 31 04:38:18.725445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.724342    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cdacda-3334-40a5-b11e-1ba6f9fb7231" containerName="revoke-keys"
Oct 31 04:38:18.727903 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.727880    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.780594 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.780561    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"]
Oct 31 04:38:18.801684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.801655    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:38:18.825556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.822733    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:38:18.825780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.825685    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:38:18.825875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.825820    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:38:18.830998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.830962    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.831183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.831071    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzjf\" (UniqueName: \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.831183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.831160    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.932070 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.932034    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzjf\" (UniqueName: \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.932295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.932128    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.932295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.932194    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.932889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.932849    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.934706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.934684    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:18.985870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:18.985804    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzjf\" (UniqueName: \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\") pod \"maas-api-revoke-sub-3e31a506-jthg4\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:19.044447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:19.044417    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:19.413164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:19.413114    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"]
Oct 31 04:38:19.415974 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:38:19.415945    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708bffef_efcb_4646_9042_8d733534fd1f.slice/crio-b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6 WatchSource:0}: Error finding container b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6: Status 404 returned error can't find the container with id b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6
Oct 31 04:38:20.285710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:20.285340    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4" event={"ID":"708bffef-efcb-4646-9042-8d733534fd1f","Type":"ContainerStarted","Data":"d5decaacc0658387f45c10f3d0e240d659499a5e8758fb07756ce9165c03ef22"}
Oct 31 04:38:20.285710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:20.285410    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4" event={"ID":"708bffef-efcb-4646-9042-8d733534fd1f","Type":"ContainerStarted","Data":"b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6"}
Oct 31 04:38:21.315556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:21.315043    5831 generic.go:356] "Generic (PLEG): container finished" podID="708bffef-efcb-4646-9042-8d733534fd1f" containerID="d5decaacc0658387f45c10f3d0e240d659499a5e8758fb07756ce9165c03ef22" exitCode=0
Oct 31 04:38:21.315556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:21.315248    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4" event={"ID":"708bffef-efcb-4646-9042-8d733534fd1f","Type":"ContainerDied","Data":"d5decaacc0658387f45c10f3d0e240d659499a5e8758fb07756ce9165c03ef22"}
Oct 31 04:38:22.506138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.506114    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:22.632086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.632015    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\") pod \"708bffef-efcb-4646-9042-8d733534fd1f\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") "
Oct 31 04:38:22.632240 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.632173    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\") pod \"708bffef-efcb-4646-9042-8d733534fd1f\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") "
Oct 31 04:38:22.632240 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.632199    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\" (UniqueName: \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\") pod \"708bffef-efcb-4646-9042-8d733534fd1f\" (UID: \"708bffef-efcb-4646-9042-8d733534fd1f\") "
Oct 31 04:38:22.632636 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.632601    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca" pod "708bffef-efcb-4646-9042-8d733534fd1f" (UID: "708bffef-efcb-4646-9042-8d733534fd1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:22.634489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.634459    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf" pod "708bffef-efcb-4646-9042-8d733534fd1f" (UID: "708bffef-efcb-4646-9042-8d733534fd1f"). InnerVolumeSpecName "kube-api-access-ltzjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:22.634721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.634522    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca" pod "708bffef-efcb-4646-9042-8d733534fd1f" (UID: "708bffef-efcb-4646-9042-8d733534fd1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:38:22.733151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.733115    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:22.733151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.733143    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzjf\" (UniqueName: \"kubernetes.io/projected/708bffef-efcb-4646-9042-8d733534fd1f-kube-api-access-ltzjf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:22.733151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:22.733153    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708bffef-efcb-4646-9042-8d733534fd1f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:23.347622 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:23.347594    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"
Oct 31 04:38:23.347857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:23.347582    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4" event={"ID":"708bffef-efcb-4646-9042-8d733534fd1f","Type":"ContainerDied","Data":"b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6"}
Oct 31 04:38:23.347857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:23.347719    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09a7444ffb9874ac0ea284545442da9186d445d9304c89b1caf0d21900886d6"
Oct 31 04:38:27.196179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:27.196144    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:38:37.002582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:37.001235    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:38:37.640126 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:37.640074    5831 scope.go:122] "RemoveContainer" containerID="20d658592d8c5e6be9b4e99fda18c2851040685ec182a16f0d25758f4dbdc49d"
Oct 31 04:38:37.654541 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:37.654362    5831 scope.go:122] "RemoveContainer" containerID="1f4a0654d12873bd18a3075f82dd6e4e2717405081900dbaf4fe04ff2f7cc573"
Oct 31 04:38:42.177223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.177189    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"]
Oct 31 04:38:42.177670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.177651    5831 cpu_manager.go:414] "containerMap: removing container" podUID="708bffef-efcb-4646-9042-8d733534fd1f" containerName="revoke-keys"
Oct 31 04:38:42.177732 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.177672    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708bffef-efcb-4646-9042-8d733534fd1f" containerName="revoke-keys"
Oct 31 04:38:42.177784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.177774    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="708bffef-efcb-4646-9042-8d733534fd1f" containerName="revoke-keys"
Oct 31 04:38:42.182660 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.182638    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.212256 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.212225    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.212420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.212261    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7czf\" (UniqueName: \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.212420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.212279    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.232162 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.232133    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:38:42.235325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.235293    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"]
Oct 31 04:38:42.277334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.277294    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:38:42.277334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.277314    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:38:42.277334 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.277297    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:38:42.312996 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.312965    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.312996 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.313000    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7czf\" (UniqueName: \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.313216 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.313018    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.313618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.313598    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.315585 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.315563    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.366164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.366130    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7czf\" (UniqueName: \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\") pod \"maas-api-revoke-sub-e89ee9a1-l7rsn\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.497306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.497273    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:42.869011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:42.868984    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"]
Oct 31 04:38:42.871124 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:38:42.871076    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4337f33_e362_4949_9b82_4ed5c865ad49.slice/crio-298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3 WatchSource:0}: Error finding container 298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3: Status 404 returned error can't find the container with id 298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3
Oct 31 04:38:42.977781 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:38:42.977745    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:38:43.549934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:43.549835    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn" event={"ID":"c4337f33-e362-4949-9b82-4ed5c865ad49","Type":"ContainerStarted","Data":"49df41273e252843403a689c695762e8b989f783a05b6fb430ac88f68ce570b6"}
Oct 31 04:38:43.549934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:43.549891    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn" event={"ID":"c4337f33-e362-4949-9b82-4ed5c865ad49","Type":"ContainerStarted","Data":"298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3"}
Oct 31 04:38:43.658222 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:43.658166    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn" podStartSLOduration=1.658125182 podStartE2EDuration="1.658125182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:43.655602634 +0000 UTC m=+5473.670164345" watchObservedRunningTime="2026-10-31 04:38:43.658125182 +0000 UTC m=+5473.672686890"
Oct 31 04:38:44.558690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:44.558655    5831 generic.go:356] "Generic (PLEG): container finished" podID="c4337f33-e362-4949-9b82-4ed5c865ad49" containerID="49df41273e252843403a689c695762e8b989f783a05b6fb430ac88f68ce570b6" exitCode=0
Oct 31 04:38:44.559146 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:44.558743    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn" event={"ID":"c4337f33-e362-4949-9b82-4ed5c865ad49","Type":"ContainerDied","Data":"49df41273e252843403a689c695762e8b989f783a05b6fb430ac88f68ce570b6"}
Oct 31 04:38:45.703651 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.703628    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:45.740865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.740836    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\") pod \"c4337f33-e362-4949-9b82-4ed5c865ad49\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") "
Oct 31 04:38:45.741017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.740882    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\" (UniqueName: \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\") pod \"c4337f33-e362-4949-9b82-4ed5c865ad49\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") "
Oct 31 04:38:45.741017 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.740949    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\") pod \"c4337f33-e362-4949-9b82-4ed5c865ad49\" (UID: \"c4337f33-e362-4949-9b82-4ed5c865ad49\") "
Oct 31 04:38:45.741338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.741311    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca" pod "c4337f33-e362-4949-9b82-4ed5c865ad49" (UID: "c4337f33-e362-4949-9b82-4ed5c865ad49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:45.743045 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.743023    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca" pod "c4337f33-e362-4949-9b82-4ed5c865ad49" (UID: "c4337f33-e362-4949-9b82-4ed5c865ad49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:38:45.743135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.743079    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf" pod "c4337f33-e362-4949-9b82-4ed5c865ad49" (UID: "c4337f33-e362-4949-9b82-4ed5c865ad49"). InnerVolumeSpecName "kube-api-access-w7czf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:45.841773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.841686    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:45.841773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.841714    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7czf\" (UniqueName: \"kubernetes.io/projected/c4337f33-e362-4949-9b82-4ed5c865ad49-kube-api-access-w7czf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:45.841773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:45.841731    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4337f33-e362-4949-9b82-4ed5c865ad49-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:46.571318 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:46.571288    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"
Oct 31 04:38:46.571503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:46.571287    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn" event={"ID":"c4337f33-e362-4949-9b82-4ed5c865ad49","Type":"ContainerDied","Data":"298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3"}
Oct 31 04:38:46.571503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:46.571420    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298abdb9f37247e6d9b06a7d9ce75d66b9fd64e6224b7333413ca05b9af2b8e3"
Oct 31 04:38:54.967052 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:38:54.967011    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:39:12.977593 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:39:12.977491    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:39:37.696125 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:39:37.696095    5831 scope.go:122] "RemoveContainer" containerID="a661c45a6f18fc67014f3d20e264ef098c361041c188f85fa443ee2ceb440854"
Oct 31 04:39:37.705964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:39:37.705944    5831 scope.go:122] "RemoveContainer" containerID="5353139b718a3d483aa3582250bf09150a55634a3d080ea56e3a5f9861713322"
Oct 31 04:39:42.978020 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:39:42.977977    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:39:48.066090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:39:48.066046    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:39:50.556738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:39:50.556693    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:39:50.557440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:39:50.557386    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:40:12.977678 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:40:12.977641    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:40:42.978613 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:40:42.978495    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:40:58.042691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:40:58.042629    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"]
Oct 31 04:40:58.088617 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:40:58.088584    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb9ef3d-r2jsz"]
Oct 31 04:40:58.133736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:40:58.133698    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974f3698-2b43-4f5f-ab8c-b57935de1d52" path="/var/lib/kubelet/pods/974f3698-2b43-4f5f-ab8c-b57935de1d52/volumes"
Oct 31 04:41:12.978944 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:41:12.978908    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:41:31.526303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.526270    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"]
Oct 31 04:41:31.526803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.526714    5831 cpu_manager.go:414] "containerMap: removing container" podUID="c4337f33-e362-4949-9b82-4ed5c865ad49" containerName="revoke-keys"
Oct 31 04:41:31.526803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.526726    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4337f33-e362-4949-9b82-4ed5c865ad49" containerName="revoke-keys"
Oct 31 04:41:31.526803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.526801    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4337f33-e362-4949-9b82-4ed5c865ad49" containerName="revoke-keys"
Oct 31 04:41:31.530273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.530254    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.578779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.578749    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:41:31.624834 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.624798    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:41:31.625053 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.624857    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:41:31.625053 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.624889    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:41:31.627248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.627224    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"]
Oct 31 04:41:31.654014 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.653980    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5c9s\" (UniqueName: \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.654167 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.654040    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.654167 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.654130    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.755661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.755626    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.755845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.755684    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.755845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.755750    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5c9s\" (UniqueName: \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.756407 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.756346    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.758224 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.758200    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.808661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.808600    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5c9s\" (UniqueName: \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\") pod \"maas-api-revoke-sub-e2547622-6z6vz\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:31.843551 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:31.843513    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:32.217967 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:32.217938    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"]
Oct 31 04:41:32.220149 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:41:32.220120    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0699f9f_1c17_4abe_a032_45b31326adc4.slice/crio-f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06 WatchSource:0}: Error finding container f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06: Status 404 returned error can't find the container with id f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06
Oct 31 04:41:32.607007 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:32.606973    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz" event={"ID":"f0699f9f-1c17-4abe-a032-45b31326adc4","Type":"ContainerStarted","Data":"32661fa6b9f145472123f14e1b8521f2d4bf813e953d3c4e675de4a24517257b"}
Oct 31 04:41:32.607411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:32.607031    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz" event={"ID":"f0699f9f-1c17-4abe-a032-45b31326adc4","Type":"ContainerStarted","Data":"f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06"}
Oct 31 04:41:32.713810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:32.713757    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz" podStartSLOduration=1.7137437709999999 podStartE2EDuration="1.713743771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:32.710852827 +0000 UTC m=+5642.725414535" watchObservedRunningTime="2026-10-31 04:41:32.713743771 +0000 UTC m=+5642.728305478"
Oct 31 04:41:33.614055 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:33.614017    5831 generic.go:356] "Generic (PLEG): container finished" podID="f0699f9f-1c17-4abe-a032-45b31326adc4" containerID="32661fa6b9f145472123f14e1b8521f2d4bf813e953d3c4e675de4a24517257b" exitCode=0
Oct 31 04:41:33.614475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:33.614102    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz" event={"ID":"f0699f9f-1c17-4abe-a032-45b31326adc4","Type":"ContainerDied","Data":"32661fa6b9f145472123f14e1b8521f2d4bf813e953d3c4e675de4a24517257b"}
Oct 31 04:41:34.759484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.759461    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:34.888103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.888008    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\" (UniqueName: \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\") pod \"f0699f9f-1c17-4abe-a032-45b31326adc4\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") "
Oct 31 04:41:34.888103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.888065    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\") pod \"f0699f9f-1c17-4abe-a032-45b31326adc4\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") "
Oct 31 04:41:34.888363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.888139    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\") pod \"f0699f9f-1c17-4abe-a032-45b31326adc4\" (UID: \"f0699f9f-1c17-4abe-a032-45b31326adc4\") "
Oct 31 04:41:34.888638 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.888605    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca" pod "f0699f9f-1c17-4abe-a032-45b31326adc4" (UID: "f0699f9f-1c17-4abe-a032-45b31326adc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:41:34.890474 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.890446    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s" pod "f0699f9f-1c17-4abe-a032-45b31326adc4" (UID: "f0699f9f-1c17-4abe-a032-45b31326adc4"). InnerVolumeSpecName "kube-api-access-d5c9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:41:34.890474 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.890463    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca" pod "f0699f9f-1c17-4abe-a032-45b31326adc4" (UID: "f0699f9f-1c17-4abe-a032-45b31326adc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:41:34.989372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.989335    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5c9s\" (UniqueName: \"kubernetes.io/projected/f0699f9f-1c17-4abe-a032-45b31326adc4-kube-api-access-d5c9s\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:34.989372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.989367    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:34.989372 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:34.989379    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0699f9f-1c17-4abe-a032-45b31326adc4-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:35.628459 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:35.628429    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"
Oct 31 04:41:35.628670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:35.628467    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz" event={"ID":"f0699f9f-1c17-4abe-a032-45b31326adc4","Type":"ContainerDied","Data":"f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06"}
Oct 31 04:41:35.628670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:35.628513    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29abadfd971e01f2193e0c12c5850657ff2cb4a0587e7c60c73caa0ba066e06"
Oct 31 04:41:42.977651 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:41:42.977611    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:41:46.176422 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:46.176390    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:41:48.051668 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:48.051625    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:41:48.052324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:48.052290    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:41:48.089556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:48.088636    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:41:48.089556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:48.088703    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:41:56.649658 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:41:56.649622    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:42:03.149101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.149010    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"]
Oct 31 04:42:03.149564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.149517    5831 cpu_manager.go:414] "containerMap: removing container" podUID="f0699f9f-1c17-4abe-a032-45b31326adc4" containerName="revoke-keys"
Oct 31 04:42:03.149564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.149552    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0699f9f-1c17-4abe-a032-45b31326adc4" containerName="revoke-keys"
Oct 31 04:42:03.149715 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.149673    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0699f9f-1c17-4abe-a032-45b31326adc4" containerName="revoke-keys"
Oct 31 04:42:03.153796 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.153775    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.202383 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.202348    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:42:03.243085 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.243048    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299bj\" (UniqueName: \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.243288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.243117    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.243364 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.243328    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.248282 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.248245    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:42:03.248436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.248256    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:42:03.248436 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.248318    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:42:03.251670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.251636    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"]
Oct 31 04:42:03.344972 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.344938    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.345150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.345039    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299bj\" (UniqueName: \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.345150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.345083    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.345849 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.345768    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.347501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.347475    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.398468 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.398440    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-299bj\" (UniqueName: \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\") pod \"maas-api-revoke-sub-c1e271a9-c77dl\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.467039 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.467008    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:03.835910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.835858    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"]
Oct 31 04:42:03.838749 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:42:03.838719    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac483155_efef_4681_9f5d_2224eb31833d.slice/crio-41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1 WatchSource:0}: Error finding container 41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1: Status 404 returned error can't find the container with id 41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1
Oct 31 04:42:03.848208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:03.848175    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl" event={"ID":"ac483155-efef-4681-9f5d-2224eb31833d","Type":"ContainerStarted","Data":"41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1"}
Oct 31 04:42:04.856490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:04.856450    5831 generic.go:356] "Generic (PLEG): container finished" podID="ac483155-efef-4681-9f5d-2224eb31833d" containerID="6aab540843d92cd315f67d1e260f3c9d2d107f833fe2d1c63285122ecc66b887" exitCode=0
Oct 31 04:42:04.856962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:04.856558    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl" event={"ID":"ac483155-efef-4681-9f5d-2224eb31833d","Type":"ContainerDied","Data":"6aab540843d92cd315f67d1e260f3c9d2d107f833fe2d1c63285122ecc66b887"}
Oct 31 04:42:06.001232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.001204    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:06.174396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.174319    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\") pod \"ac483155-efef-4681-9f5d-2224eb31833d\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") "
Oct 31 04:42:06.174396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.174372    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\") pod \"ac483155-efef-4681-9f5d-2224eb31833d\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") "
Oct 31 04:42:06.174635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.174421    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\" (UniqueName: \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\") pod \"ac483155-efef-4681-9f5d-2224eb31833d\" (UID: \"ac483155-efef-4681-9f5d-2224eb31833d\") "
Oct 31 04:42:06.174811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.174786    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca" pod "ac483155-efef-4681-9f5d-2224eb31833d" (UID: "ac483155-efef-4681-9f5d-2224eb31833d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:06.176788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.176762    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj" pod "ac483155-efef-4681-9f5d-2224eb31833d" (UID: "ac483155-efef-4681-9f5d-2224eb31833d"). InnerVolumeSpecName "kube-api-access-299bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:06.176788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.176773    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca" pod "ac483155-efef-4681-9f5d-2224eb31833d" (UID: "ac483155-efef-4681-9f5d-2224eb31833d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:42:06.275242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.275209    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:06.275242 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.275237    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-299bj\" (UniqueName: \"kubernetes.io/projected/ac483155-efef-4681-9f5d-2224eb31833d-kube-api-access-299bj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:06.275468 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.275251    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac483155-efef-4681-9f5d-2224eb31833d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:06.869920 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.869886    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"
Oct 31 04:42:06.870110 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.869934    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl" event={"ID":"ac483155-efef-4681-9f5d-2224eb31833d","Type":"ContainerDied","Data":"41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1"}
Oct 31 04:42:06.870110 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:06.869986    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41467f5a3fe95a9e2da01d7be91e66fa56c696b77c16fa216faa0cba7b4b29c1"
Oct 31 04:42:12.978050 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:42:12.978015    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:42:15.950861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:15.950822    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:42:26.896457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:26.896414    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:42:28.284520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.284481    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"]
Oct 31 04:42:28.284968 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.284954    5831 cpu_manager.go:414] "containerMap: removing container" podUID="ac483155-efef-4681-9f5d-2224eb31833d" containerName="revoke-keys"
Oct 31 04:42:28.285020 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.284970    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac483155-efef-4681-9f5d-2224eb31833d" containerName="revoke-keys"
Oct 31 04:42:28.285066 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.285042    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac483155-efef-4681-9f5d-2224eb31833d" containerName="revoke-keys"
Oct 31 04:42:28.288394 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.288374    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.371739 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.371689    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc44c\" (UniqueName: \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.371927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.371756    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.371927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.371855    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.383989 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.383963    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:42:28.384185 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.384008    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:42:28.384300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.384049    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:42:28.384300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.384058    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:42:28.386575 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.386555    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"]
Oct 31 04:42:28.472484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.472450    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc44c\" (UniqueName: \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.472710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.472512    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.472710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.472577    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.473200 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.473174    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.475099 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.475080    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.525390 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.525359    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc44c\" (UniqueName: \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\") pod \"maas-api-revoke-sub-2e1f5b5f-w6gqf\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.601694 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.601615    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:28.971179 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:28.971153    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"]
Oct 31 04:42:28.974584 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:42:28.974558    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecba983_4cb9_4b84_8d89_418a8ca2e616.slice/crio-1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f WatchSource:0}: Error finding container 1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f: Status 404 returned error can't find the container with id 1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f
Oct 31 04:42:29.016336 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:29.015945    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf" event={"ID":"8ecba983-4cb9-4b84-8d89-418a8ca2e616","Type":"ContainerStarted","Data":"1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f"}
Oct 31 04:42:30.024273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:30.024235    5831 generic.go:356] "Generic (PLEG): container finished" podID="8ecba983-4cb9-4b84-8d89-418a8ca2e616" containerID="d3be08ec4df4c4a4c44a3231905d997c15e1146112db96bb2c02b2fdbe8b005e" exitCode=0
Oct 31 04:42:30.024706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:30.024328    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf" event={"ID":"8ecba983-4cb9-4b84-8d89-418a8ca2e616","Type":"ContainerDied","Data":"d3be08ec4df4c4a4c44a3231905d997c15e1146112db96bb2c02b2fdbe8b005e"}
Oct 31 04:42:31.175931 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.175904    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:31.301068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.300990    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\") pod \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") "
Oct 31 04:42:31.301248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.301092    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\" (UniqueName: \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\") pod \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") "
Oct 31 04:42:31.301248 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.301127    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\") pod \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\" (UID: \"8ecba983-4cb9-4b84-8d89-418a8ca2e616\") "
Oct 31 04:42:31.301396 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.301368    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca" pod "8ecba983-4cb9-4b84-8d89-418a8ca2e616" (UID: "8ecba983-4cb9-4b84-8d89-418a8ca2e616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:31.303475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.303425    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca" pod "8ecba983-4cb9-4b84-8d89-418a8ca2e616" (UID: "8ecba983-4cb9-4b84-8d89-418a8ca2e616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:42:31.303811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.303784    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c" pod "8ecba983-4cb9-4b84-8d89-418a8ca2e616" (UID: "8ecba983-4cb9-4b84-8d89-418a8ca2e616"). InnerVolumeSpecName "kube-api-access-bc44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:31.402864 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.402827    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:31.402864 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.402863    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ecba983-4cb9-4b84-8d89-418a8ca2e616-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:31.403083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:31.402877    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc44c\" (UniqueName: \"kubernetes.io/projected/8ecba983-4cb9-4b84-8d89-418a8ca2e616-kube-api-access-bc44c\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:32.039408 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:32.039375    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"
Oct 31 04:42:32.039678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:32.039378    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf" event={"ID":"8ecba983-4cb9-4b84-8d89-418a8ca2e616","Type":"ContainerDied","Data":"1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f"}
Oct 31 04:42:32.039678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:32.039500    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bba288a7baf8ac68fe1adab8a6c9790bea918c2c898be75918787dcd029ef0f"
Oct 31 04:42:37.756356 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:37.756323    5831 scope.go:122] "RemoveContainer" containerID="68c62049acfa514c79ee2836dd80dc2d464f8299cecfc363dafabc52b94b669a"
Oct 31 04:42:38.871118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.871087    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:42:38.879621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.879598    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:42:38.881160 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.881139    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:42:38.884889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.884868    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:42:38.889409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.889386    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:42:38.892964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:38.892946    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:42:40.553488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.553449    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"]
Oct 31 04:42:40.554223 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.554198    5831 cpu_manager.go:414] "containerMap: removing container" podUID="8ecba983-4cb9-4b84-8d89-418a8ca2e616" containerName="revoke-keys"
Oct 31 04:42:40.554331 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.554225    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecba983-4cb9-4b84-8d89-418a8ca2e616" containerName="revoke-keys"
Oct 31 04:42:40.554390 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.554377    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecba983-4cb9-4b84-8d89-418a8ca2e616" containerName="revoke-keys"
Oct 31 04:42:40.559211 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.559191    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.608875 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.608843    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"]
Oct 31 04:42:40.650487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.650454    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:42:40.650487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.650465    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:42:40.650724 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.650484    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:42:40.652845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.652829    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:42:40.703331 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.703302    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd86n\" (UniqueName: \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.703473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.703364    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.703473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.703436    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.804690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.804603    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.804690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.804683    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd86n\" (UniqueName: \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.804956 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.804757    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.805289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.805264    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.807294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.807272    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.857473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.857442    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd86n\" (UniqueName: \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\") pod \"maas-api-revoke-sub-ec843018-498qh\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:40.875208 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:40.875185    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:41.243895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:41.243713    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"]
Oct 31 04:42:41.246511 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:42:41.246480    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4a6bde_923b_48bb_adec_5954b57daeba.slice/crio-29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141 WatchSource:0}: Error finding container 29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141: Status 404 returned error can't find the container with id 29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141
Oct 31 04:42:42.096614 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:42.096581    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh" event={"ID":"6e4a6bde-923b-48bb-adec-5954b57daeba","Type":"ContainerStarted","Data":"973fa66222a3c379b041dd4dd8f362021f3c63e4588f1188dbff757ca09c4c18"}
Oct 31 04:42:42.097022 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:42.096633    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh" event={"ID":"6e4a6bde-923b-48bb-adec-5954b57daeba","Type":"ContainerStarted","Data":"29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141"}
Oct 31 04:42:42.212779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:42.212731    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh" podStartSLOduration=2.212716573 podStartE2EDuration="2.212716573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:42.212318733 +0000 UTC m=+5712.226880443" watchObservedRunningTime="2026-10-31 04:42:42.212716573 +0000 UTC m=+5712.227278280"
Oct 31 04:42:42.977858 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:42:42.977821    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:42:43.105761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:43.105725    5831 generic.go:356] "Generic (PLEG): container finished" podID="6e4a6bde-923b-48bb-adec-5954b57daeba" containerID="973fa66222a3c379b041dd4dd8f362021f3c63e4588f1188dbff757ca09c4c18" exitCode=0
Oct 31 04:42:43.106194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:43.105822    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh" event={"ID":"6e4a6bde-923b-48bb-adec-5954b57daeba","Type":"ContainerDied","Data":"973fa66222a3c379b041dd4dd8f362021f3c63e4588f1188dbff757ca09c4c18"}
Oct 31 04:42:44.264466 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.264441    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:44.441520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.441490    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\") pod \"6e4a6bde-923b-48bb-adec-5954b57daeba\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") "
Oct 31 04:42:44.441727 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.441641    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\" (UniqueName: \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\") pod \"6e4a6bde-923b-48bb-adec-5954b57daeba\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") "
Oct 31 04:42:44.441794 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.441779    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\") pod \"6e4a6bde-923b-48bb-adec-5954b57daeba\" (UID: \"6e4a6bde-923b-48bb-adec-5954b57daeba\") "
Oct 31 04:42:44.442155 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.442126    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca" pod "6e4a6bde-923b-48bb-adec-5954b57daeba" (UID: "6e4a6bde-923b-48bb-adec-5954b57daeba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:44.443851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.443824    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca" pod "6e4a6bde-923b-48bb-adec-5954b57daeba" (UID: "6e4a6bde-923b-48bb-adec-5954b57daeba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:42:44.443960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.443912    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n" pod "6e4a6bde-923b-48bb-adec-5954b57daeba" (UID: "6e4a6bde-923b-48bb-adec-5954b57daeba"). InnerVolumeSpecName "kube-api-access-wd86n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:44.542775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.542740    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:44.542775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.542772    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a6bde-923b-48bb-adec-5954b57daeba-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:44.542775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.542784    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd86n\" (UniqueName: \"kubernetes.io/projected/6e4a6bde-923b-48bb-adec-5954b57daeba-kube-api-access-wd86n\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:42:44.606522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:44.606489    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:42:45.121969 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:45.121938    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"
Oct 31 04:42:45.122164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:45.121932    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh" event={"ID":"6e4a6bde-923b-48bb-adec-5954b57daeba","Type":"ContainerDied","Data":"29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141"}
Oct 31 04:42:45.122164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:45.122073    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b12afdc91bb81320fcb777a7b698c0c4257a0d40e7ceb6ac85b6d4a5194141"
Oct 31 04:42:59.298810 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:42:59.298771    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:43:02.612553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:02.611645    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:43:02.612553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:02.611725    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:43:03.057103 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:03.057062    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"]
Oct 31 04:43:03.102501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:03.102464    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4p85w"]
Oct 31 04:43:04.136650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:04.136617    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29cdacda-3334-40a5-b11e-1ba6f9fb7231" path="/var/lib/kubelet/pods/29cdacda-3334-40a5-b11e-1ba6f9fb7231/volumes"
Oct 31 04:43:12.977487 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:43:12.977449    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:43:18.705998 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.705959    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"]
Oct 31 04:43:18.706434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.706417    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6e4a6bde-923b-48bb-adec-5954b57daeba" containerName="revoke-keys"
Oct 31 04:43:18.706496 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.706437    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4a6bde-923b-48bb-adec-5954b57daeba" containerName="revoke-keys"
Oct 31 04:43:18.706567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.706546    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4a6bde-923b-48bb-adec-5954b57daeba" containerName="revoke-keys"
Oct 31 04:43:18.710194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.710173    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.804685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.804647    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:43:18.804685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.804663    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:43:18.804986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.804770    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:43:18.804986 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.804855    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:43:18.807360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.807332    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"]
Oct 31 04:43:18.865979 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.865947    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.866148 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.866106    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.866196 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.866175    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672r9\" (UniqueName: \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.967705 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.967612    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.967705 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.967670    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672r9\" (UniqueName: \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.967938 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.967715    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.968263 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.968243    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:18.970310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:18.970288    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:19.020428 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:19.020392    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672r9\" (UniqueName: \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\") pod \"maas-api-revoke-sub-046e1343-m75df\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:19.025238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:19.025215    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:19.393175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:19.393122    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"]
Oct 31 04:43:19.395797 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:43:19.395763    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5977949b_9cf9_48aa_89c3_13ff09a9551c.slice/crio-257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31 WatchSource:0}: Error finding container 257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31: Status 404 returned error can't find the container with id 257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31
Oct 31 04:43:19.456079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:19.456044    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df" event={"ID":"5977949b-9cf9-48aa-89c3-13ff09a9551c","Type":"ContainerStarted","Data":"257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31"}
Oct 31 04:43:20.463303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:20.463267    5831 generic.go:356] "Generic (PLEG): container finished" podID="5977949b-9cf9-48aa-89c3-13ff09a9551c" containerID="e10c73a1e74680c3896ca4cd72296a64cb5182b4c92219a9d86c6c49769344d1" exitCode=0
Oct 31 04:43:20.463303 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:20.463307    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df" event={"ID":"5977949b-9cf9-48aa-89c3-13ff09a9551c","Type":"ContainerDied","Data":"e10c73a1e74680c3896ca4cd72296a64cb5182b4c92219a9d86c6c49769344d1"}
Oct 31 04:43:21.612166 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.612137    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:21.796873 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.796782    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\") pod \"5977949b-9cf9-48aa-89c3-13ff09a9551c\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") "
Oct 31 04:43:21.797060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.796881    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\") pod \"5977949b-9cf9-48aa-89c3-13ff09a9551c\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") "
Oct 31 04:43:21.797060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.796973    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\" (UniqueName: \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\") pod \"5977949b-9cf9-48aa-89c3-13ff09a9551c\" (UID: \"5977949b-9cf9-48aa-89c3-13ff09a9551c\") "
Oct 31 04:43:21.797273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.797252    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca" pod "5977949b-9cf9-48aa-89c3-13ff09a9551c" (UID: "5977949b-9cf9-48aa-89c3-13ff09a9551c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:43:21.797385 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.797371    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:21.799374 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.799349    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca" pod "5977949b-9cf9-48aa-89c3-13ff09a9551c" (UID: "5977949b-9cf9-48aa-89c3-13ff09a9551c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:43:21.799490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.799373    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9" pod "5977949b-9cf9-48aa-89c3-13ff09a9551c" (UID: "5977949b-9cf9-48aa-89c3-13ff09a9551c"). InnerVolumeSpecName "kube-api-access-672r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:43:21.898490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.898458    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672r9\" (UniqueName: \"kubernetes.io/projected/5977949b-9cf9-48aa-89c3-13ff09a9551c-kube-api-access-672r9\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:21.898490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:21.898489    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5977949b-9cf9-48aa-89c3-13ff09a9551c-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:22.477222 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:22.477186    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df" event={"ID":"5977949b-9cf9-48aa-89c3-13ff09a9551c","Type":"ContainerDied","Data":"257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31"}
Oct 31 04:43:22.477419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:22.477248    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257cf4c0d4b1510be7e9ac8d376f8b8baae13d13f71453a64dff6d316ef19b31"
Oct 31 04:43:22.477419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:22.477198    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"
Oct 31 04:43:23.042905 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:23.042875    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"]
Oct 31 04:43:23.045106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:23.045079    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jthg4"]
Oct 31 04:43:24.134481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:24.134448    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708bffef-efcb-4646-9042-8d733534fd1f" path="/var/lib/kubelet/pods/708bffef-efcb-4646-9042-8d733534fd1f/volumes"
Oct 31 04:43:29.408041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.407961    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"]
Oct 31 04:43:29.408453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.408439    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5977949b-9cf9-48aa-89c3-13ff09a9551c" containerName="revoke-keys"
Oct 31 04:43:29.408453 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.408452    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5977949b-9cf9-48aa-89c3-13ff09a9551c" containerName="revoke-keys"
Oct 31 04:43:29.408551 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.408540    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5977949b-9cf9-48aa-89c3-13ff09a9551c" containerName="revoke-keys"
Oct 31 04:43:29.412273 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.412254    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.461021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.460987    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:43:29.463860 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.463829    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"]
Oct 31 04:43:29.467625 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.467600    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rxx\" (UniqueName: \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.467747 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.467730    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.467808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.467792    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.506895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.506856    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:43:29.506895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.506872    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:43:29.507084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.506892    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:43:29.568804 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.568773    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.568958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.568828    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.568958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.568863    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rxx\" (UniqueName: \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.569632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.569573    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.571398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.571374    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.622403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.622367    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rxx\" (UniqueName: \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\") pod \"maas-api-revoke-sub-d3fba056-bdhf8\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:29.725627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:29.725596    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:30.107227 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:30.107189    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"]
Oct 31 04:43:30.108705 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:43:30.108674    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef3081a_7086_4d1b_9a65_eb364f5d57cc.slice/crio-07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9 WatchSource:0}: Error finding container 07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9: Status 404 returned error can't find the container with id 07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9
Oct 31 04:43:30.527009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:30.526972    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8" event={"ID":"1ef3081a-7086-4d1b-9a65-eb364f5d57cc","Type":"ContainerStarted","Data":"730ccb855cc9078a18b8c84338a84e033fb263f12ea2738d48ac26f049439fc4"}
Oct 31 04:43:30.527563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:30.527029    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8" event={"ID":"1ef3081a-7086-4d1b-9a65-eb364f5d57cc","Type":"ContainerStarted","Data":"07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9"}
Oct 31 04:43:31.496915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:31.496857    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8" podStartSLOduration=2.496842851 podStartE2EDuration="2.496842851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:43:30.678409406 +0000 UTC m=+5760.692971114" watchObservedRunningTime="2026-10-31 04:43:31.496842851 +0000 UTC m=+5761.511404605"
Oct 31 04:43:31.497119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:31.497069    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:43:31.536278 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:31.536239    5831 generic.go:356] "Generic (PLEG): container finished" podID="1ef3081a-7086-4d1b-9a65-eb364f5d57cc" containerID="730ccb855cc9078a18b8c84338a84e033fb263f12ea2738d48ac26f049439fc4" exitCode=0
Oct 31 04:43:31.536762 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:31.536321    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8" event={"ID":"1ef3081a-7086-4d1b-9a65-eb364f5d57cc","Type":"ContainerDied","Data":"730ccb855cc9078a18b8c84338a84e033fb263f12ea2738d48ac26f049439fc4"}
Oct 31 04:43:32.773358 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.773312    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:32.808604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.806858    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\" (UniqueName: \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\") pod \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") "
Oct 31 04:43:32.808604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.807068    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\") pod \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") "
Oct 31 04:43:32.808604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.807104    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\") pod \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\" (UID: \"1ef3081a-7086-4d1b-9a65-eb364f5d57cc\") "
Oct 31 04:43:32.808604 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.807796    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca" pod "1ef3081a-7086-4d1b-9a65-eb364f5d57cc" (UID: "1ef3081a-7086-4d1b-9a65-eb364f5d57cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:43:32.843552 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.842772    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx" pod "1ef3081a-7086-4d1b-9a65-eb364f5d57cc" (UID: "1ef3081a-7086-4d1b-9a65-eb364f5d57cc"). InnerVolumeSpecName "kube-api-access-r2rxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:43:32.851553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.845699    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca" pod "1ef3081a-7086-4d1b-9a65-eb364f5d57cc" (UID: "1ef3081a-7086-4d1b-9a65-eb364f5d57cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:43:32.908553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.908368    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:32.908553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.908405    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:32.908553 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:32.908424    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2rxx\" (UniqueName: \"kubernetes.io/projected/1ef3081a-7086-4d1b-9a65-eb364f5d57cc-kube-api-access-r2rxx\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:43:33.580555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:33.577869    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8" event={"ID":"1ef3081a-7086-4d1b-9a65-eb364f5d57cc","Type":"ContainerDied","Data":"07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9"}
Oct 31 04:43:33.580555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:33.577936    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07f1a1243862c922fb2e82bf96b37ae357b7fd8407c7ecb614c15d68df5705f9"
Oct 31 04:43:33.580555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:33.578009    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"
Oct 31 04:43:42.192557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:42.192512    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:43:42.979051 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:43:42.979010    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:43:46.043635 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:46.043605    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"]
Oct 31 04:43:46.089973 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:46.089943    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e89ee9a1-l7rsn"]
Oct 31 04:43:46.133522 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:46.133492    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4337f33-e362-4949-9b82-4ed5c865ad49" path="/var/lib/kubelet/pods/c4337f33-e362-4949-9b82-4ed5c865ad49/volumes"
Oct 31 04:43:54.654106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:43:54.654073    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:03.859501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.859466    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"]
Oct 31 04:44:03.860064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.859903    5831 cpu_manager.go:414] "containerMap: removing container" podUID="1ef3081a-7086-4d1b-9a65-eb364f5d57cc" containerName="revoke-keys"
Oct 31 04:44:03.860064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.859914    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef3081a-7086-4d1b-9a65-eb364f5d57cc" containerName="revoke-keys"
Oct 31 04:44:03.860064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.860004    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef3081a-7086-4d1b-9a65-eb364f5d57cc" containerName="revoke-keys"
Oct 31 04:44:03.864415 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.864394    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:03.913286 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.913246    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:44:03.916520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.916488    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"]
Oct 31 04:44:03.933936 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.933908    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:03.934106 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.933984    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq87q\" (UniqueName: \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:03.934173 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.934092    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:03.959003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.958966    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:44:03.959003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.958986    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:44:03.959003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:03.958986    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:44:04.035193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.035160    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq87q\" (UniqueName: \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.035193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.035199    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.035419 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.035247    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.035889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.035868    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.037927 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.037900    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.088172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.088147    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq87q\" (UniqueName: \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\") pod \"maas-api-revoke-sub-ae594979-rswvl\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.177730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.177642    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:04.545123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.545097    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"]
Oct 31 04:44:04.548289 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:44:04.548257    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb735ec5_6c3c_42a1_987a_357b4c91655d.slice/crio-0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a WatchSource:0}: Error finding container 0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a: Status 404 returned error can't find the container with id 0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a
Oct 31 04:44:04.822135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.822061    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl" event={"ID":"cb735ec5-6c3c-42a1-987a-357b4c91655d","Type":"ContainerStarted","Data":"098da48024be8a22b160830ddd8694b641929ed1b3d2d4c9a2b84b653ea41b68"}
Oct 31 04:44:04.822135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.822120    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl" event={"ID":"cb735ec5-6c3c-42a1-987a-357b4c91655d","Type":"ContainerStarted","Data":"0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a"}
Oct 31 04:44:04.930213 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:04.930156    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl" podStartSLOduration=1.930140591 podStartE2EDuration="1.930140591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:04.927933392 +0000 UTC m=+5794.942495140" watchObservedRunningTime="2026-10-31 04:44:04.930140591 +0000 UTC m=+5794.944702296"
Oct 31 04:44:05.828959 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:05.828923    5831 generic.go:356] "Generic (PLEG): container finished" podID="cb735ec5-6c3c-42a1-987a-357b4c91655d" containerID="098da48024be8a22b160830ddd8694b641929ed1b3d2d4c9a2b84b653ea41b68" exitCode=0
Oct 31 04:44:05.829151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:05.829018    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl" event={"ID":"cb735ec5-6c3c-42a1-987a-357b4c91655d","Type":"ContainerDied","Data":"098da48024be8a22b160830ddd8694b641929ed1b3d2d4c9a2b84b653ea41b68"}
Oct 31 04:44:06.981520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:06.981494    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:07.063324 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.063289    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\") pod \"cb735ec5-6c3c-42a1-987a-357b4c91655d\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") "
Oct 31 04:44:07.063563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.063385    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\") pod \"cb735ec5-6c3c-42a1-987a-357b4c91655d\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") "
Oct 31 04:44:07.063563 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.063415    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\" (UniqueName: \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\") pod \"cb735ec5-6c3c-42a1-987a-357b4c91655d\" (UID: \"cb735ec5-6c3c-42a1-987a-357b4c91655d\") "
Oct 31 04:44:07.063831 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.063797    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca" pod "cb735ec5-6c3c-42a1-987a-357b4c91655d" (UID: "cb735ec5-6c3c-42a1-987a-357b4c91655d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:07.065841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.065817    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q" pod "cb735ec5-6c3c-42a1-987a-357b4c91655d" (UID: "cb735ec5-6c3c-42a1-987a-357b4c91655d"). InnerVolumeSpecName "kube-api-access-xq87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:07.065948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.065923    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca" pod "cb735ec5-6c3c-42a1-987a-357b4c91655d" (UID: "cb735ec5-6c3c-42a1-987a-357b4c91655d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:44:07.164395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.164287    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:07.164395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.164327    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq87q\" (UniqueName: \"kubernetes.io/projected/cb735ec5-6c3c-42a1-987a-357b4c91655d-kube-api-access-xq87q\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:07.164395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.164341    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb735ec5-6c3c-42a1-987a-357b4c91655d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:07.842376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.842345    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"
Oct 31 04:44:07.842376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.842368    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl" event={"ID":"cb735ec5-6c3c-42a1-987a-357b4c91655d","Type":"ContainerDied","Data":"0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a"}
Oct 31 04:44:07.842633 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:07.842409    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bec4a5f22c702c2a9fc2d332fc1096d5584a67eadbd37ab6e167f5b786f790a"
Oct 31 04:44:12.977605 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:44:12.977572    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:44:16.499089 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:16.499048    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:22.477503 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.477463    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"]
Oct 31 04:44:22.478835 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.478806    5831 cpu_manager.go:414] "containerMap: removing container" podUID="cb735ec5-6c3c-42a1-987a-357b4c91655d" containerName="revoke-keys"
Oct 31 04:44:22.479014 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.478997    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb735ec5-6c3c-42a1-987a-357b4c91655d" containerName="revoke-keys"
Oct 31 04:44:22.479281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.479264    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb735ec5-6c3c-42a1-987a-357b4c91655d" containerName="revoke-keys"
Oct 31 04:44:22.497341 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.497312    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.519417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.519089    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jzw\" (UniqueName: \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.519417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.519171    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.519417 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.519258    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.541349 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.541319    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"]
Oct 31 04:44:22.600501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.600458    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:44:22.600960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.600941    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:44:22.601227 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.601210    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:44:22.608741 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.608723    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:44:22.638181 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.638149    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.638808 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.638511    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.639049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.639028    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jzw\" (UniqueName: \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.657555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.640626    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.755608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.752285    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jzw\" (UniqueName: \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.806812 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.798605    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-khstn\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:22.902038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:22.901709    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:24.071029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:24.070994    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"]
Oct 31 04:44:24.141597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:24.141567    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:24.368293 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:24.368251    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn" event={"ID":"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc","Type":"ContainerStarted","Data":"219a20280dc920c4b73889356ab7ee50a1683d7e1f260a272198b0bcae517b75"}
Oct 31 04:44:25.375670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:25.375632    5831 generic.go:356] "Generic (PLEG): container finished" podID="5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" containerID="94cd423013d9cfa631a4315ba6815e5d06f7c2e59dcb377cdeea214ecfefc9c2" exitCode=0
Oct 31 04:44:25.376141 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:25.375709    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn" event={"ID":"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc","Type":"ContainerDied","Data":"94cd423013d9cfa631a4315ba6815e5d06f7c2e59dcb377cdeea214ecfefc9c2"}
Oct 31 04:44:26.521745 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.521721    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:26.602083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.602054    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\") pod \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") "
Oct 31 04:44:26.602270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.602135    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\") pod \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") "
Oct 31 04:44:26.602270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.602217    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\" (UniqueName: \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\") pod \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\" (UID: \"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc\") "
Oct 31 04:44:26.602519 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.602489    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca" pod "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" (UID: "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:26.604431 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.604406    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca" pod "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" (UID: "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:44:26.604520 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.604430    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw" pod "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" (UID: "5b43724e-e4b5-4f8d-a543-d5fee0f09fbc"). InnerVolumeSpecName "kube-api-access-64jzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:26.703212 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.703184    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:26.703212 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.703210    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jzw\" (UniqueName: \"kubernetes.io/projected/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-kube-api-access-64jzw\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:26.703437 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:26.703222    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:27.388631 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:27.388602    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"
Oct 31 04:44:27.388815 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:27.388602    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn" event={"ID":"5b43724e-e4b5-4f8d-a543-d5fee0f09fbc","Type":"ContainerDied","Data":"219a20280dc920c4b73889356ab7ee50a1683d7e1f260a272198b0bcae517b75"}
Oct 31 04:44:27.388815 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:27.388732    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="219a20280dc920c4b73889356ab7ee50a1683d7e1f260a272198b0bcae517b75"
Oct 31 04:44:32.268457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:32.268421    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:37.795289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:37.795253    5831 scope.go:122] "RemoveContainer" containerID="af80627d51d365a95be91fd4375de9b72c2b49e1cc175340ffa4b4dc182293d1"
Oct 31 04:44:37.805590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:37.805571    5831 scope.go:122] "RemoveContainer" containerID="d5decaacc0658387f45c10f3d0e240d659499a5e8758fb07756ce9165c03ef22"
Oct 31 04:44:40.695981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:40.695942    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:43.080228 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:44:43.080183    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:44:43.660556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.659921    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"]
Oct 31 04:44:43.660788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.660632    5831 cpu_manager.go:414] "containerMap: removing container" podUID="5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" containerName="revoke-keys"
Oct 31 04:44:43.660788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.660652    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" containerName="revoke-keys"
Oct 31 04:44:43.660788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.660780    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" containerName="revoke-keys"
Oct 31 04:44:43.669175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.669145    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.743591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.718959    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:44:43.743591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.719590    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:44:43.743591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.740701    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"]
Oct 31 04:44:43.783556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.762812    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:44:43.783556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.762889    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:44:43.841470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.841097    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.841470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.841168    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.841470 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.841316    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2px\" (UniqueName: \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.945556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.943187    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2px\" (UniqueName: \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.945556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.943300    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.945556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.943363    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.969571 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.966302    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:43.986588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:43.986551    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:44.027683 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:44.027598    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2px\" (UniqueName: \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\") pod \"maas-api-revoke-sub-79805ca9-gvp66\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:44.175123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:44.175087    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:44.548005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:44.547931    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"]
Oct 31 04:44:44.551580 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:44:44.551544    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd09104_ac1c_4a52_a325_23a954e30eed.slice/crio-a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d WatchSource:0}: Error finding container a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d: Status 404 returned error can't find the container with id a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d
Oct 31 04:44:45.537451 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:45.537417    5831 generic.go:356] "Generic (PLEG): container finished" podID="add09104-ac1c-4a52-a325-23a954e30eed" containerID="88e07f6cf5af42f003d741f912c2282db78872555af62a63dd17c3314d9940ec" exitCode=0
Oct 31 04:44:45.537915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:45.537501    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66" event={"ID":"add09104-ac1c-4a52-a325-23a954e30eed","Type":"ContainerDied","Data":"88e07f6cf5af42f003d741f912c2282db78872555af62a63dd17c3314d9940ec"}
Oct 31 04:44:45.537915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:45.537571    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66" event={"ID":"add09104-ac1c-4a52-a325-23a954e30eed","Type":"ContainerStarted","Data":"a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d"}
Oct 31 04:44:46.699842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.699815    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:46.772776 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.772740    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\") pod \"add09104-ac1c-4a52-a325-23a954e30eed\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") "
Oct 31 04:44:46.772982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.772916    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\") pod \"add09104-ac1c-4a52-a325-23a954e30eed\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") "
Oct 31 04:44:46.772982 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.772959    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\" (UniqueName: \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\") pod \"add09104-ac1c-4a52-a325-23a954e30eed\" (UID: \"add09104-ac1c-4a52-a325-23a954e30eed\") "
Oct 31 04:44:46.773269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.773248    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca" pod "add09104-ac1c-4a52-a325-23a954e30eed" (UID: "add09104-ac1c-4a52-a325-23a954e30eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:46.775021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.774995    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca" pod "add09104-ac1c-4a52-a325-23a954e30eed" (UID: "add09104-ac1c-4a52-a325-23a954e30eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:44:46.775147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.775118    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px" pod "add09104-ac1c-4a52-a325-23a954e30eed" (UID: "add09104-ac1c-4a52-a325-23a954e30eed"). InnerVolumeSpecName "kube-api-access-2z2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:46.874361 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.874276    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:46.874361 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.874307    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add09104-ac1c-4a52-a325-23a954e30eed-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:46.874361 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:46.874346    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2px\" (UniqueName: \"kubernetes.io/projected/add09104-ac1c-4a52-a325-23a954e30eed-kube-api-access-2z2px\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:44:47.552880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:47.552850    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"
Oct 31 04:44:47.553076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:47.552894    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66" event={"ID":"add09104-ac1c-4a52-a325-23a954e30eed","Type":"ContainerDied","Data":"a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d"}
Oct 31 04:44:47.553076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:47.552943    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45c733e0222c5f729b67d253555357896450772862c02ed7f60f8e70864830d"
Oct 31 04:44:50.974412 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:50.974378    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:44:59.374846 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:44:59.374759    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:45:00.154851 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.154812    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:00.155814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.155786    5831 cpu_manager.go:414] "containerMap: removing container" podUID="add09104-ac1c-4a52-a325-23a954e30eed" containerName="revoke-keys"
Oct 31 04:45:00.155814 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.155814    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add09104-ac1c-4a52-a325-23a954e30eed" containerName="revoke-keys"
Oct 31 04:45:00.156041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.155960    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="add09104-ac1c-4a52-a325-23a954e30eed" containerName="revoke-keys"
Oct 31 04:45:00.159790 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.159707    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.215672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.209854    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:00.215672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.213808    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklm7\" (UniqueName: \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.215672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.213957    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.255561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.252091    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:45:00.255561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.252124    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:45:00.255561 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.252263    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:45:00.314981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.314944    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklm7\" (UniqueName: \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.315300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.315271    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.316326 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.316297    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.368484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.368446    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklm7\" (UniqueName: \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\") pod \"maas-api-key-cleanup-29890365-sttbp\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:00.481001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:00.480961    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:01.069975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:01.069763    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:01.684366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:01.684311    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" event={"ID":"bd0e19e2-ff75-4dc2-9f42-558dde66113e","Type":"ContainerStarted","Data":"afcb7343470f5d63de2e1cf3a207a4e3c8b8604044cfd5f8b04f78a68fe297bf"}
Oct 31 04:45:02.720557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:02.709095    5831 generic.go:356] "Generic (PLEG): container finished" podID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerID="e37fbeb44836cbb453b24425ffaef9b909e086c7a5ce4ffad924c29ee321a0f8" exitCode=60
Oct 31 04:45:02.720557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:02.709276    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" event={"ID":"bd0e19e2-ff75-4dc2-9f42-558dde66113e","Type":"ContainerDied","Data":"e37fbeb44836cbb453b24425ffaef9b909e086c7a5ce4ffad924c29ee321a0f8"}
Oct 31 04:45:02.720557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:02.709698    5831 scope.go:122] "RemoveContainer" containerID="e37fbeb44836cbb453b24425ffaef9b909e086c7a5ce4ffad924c29ee321a0f8"
Oct 31 04:45:03.718062 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:03.718027    5831 generic.go:356] "Generic (PLEG): container finished" podID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerID="7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d" exitCode=60
Oct 31 04:45:03.718277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:03.718113    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" event={"ID":"bd0e19e2-ff75-4dc2-9f42-558dde66113e","Type":"ContainerDied","Data":"7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d"}
Oct 31 04:45:03.718277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:03.718178    5831 scope.go:122] "RemoveContainer" containerID="e37fbeb44836cbb453b24425ffaef9b909e086c7a5ce4ffad924c29ee321a0f8"
Oct 31 04:45:03.718629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:03.718609    5831 scope.go:122] "RemoveContainer" containerID="7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d"
Oct 31 04:45:03.718921 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:45:03.718900    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-sttbp_odh-ai-gateway-infra(bd0e19e2-ff75-4dc2-9f42-558dde66113e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e"
Oct 31 04:45:04.725478 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:04.725440    5831 scope.go:122] "RemoveContainer" containerID="7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d"
Oct 31 04:45:04.725912 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:45:04.725685    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-sttbp_odh-ai-gateway-infra(bd0e19e2-ff75-4dc2-9f42-558dde66113e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e"
Oct 31 04:45:08.476699 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:08.476655    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:45:12.978832 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:45:12.978722    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:45:18.127837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:18.127801    5831 scope.go:122] "RemoveContainer" containerID="7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d"
Oct 31 04:45:18.817095 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:18.817065    5831 generic.go:356] "Generic (PLEG): container finished" podID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerID="72a0ad596bbeb4363ba5f2f83ef5562204aec41c67107a1f13ae3b303377fbdf" exitCode=60
Oct 31 04:45:18.817249 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:18.817133    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" event={"ID":"bd0e19e2-ff75-4dc2-9f42-558dde66113e","Type":"ContainerDied","Data":"72a0ad596bbeb4363ba5f2f83ef5562204aec41c67107a1f13ae3b303377fbdf"}
Oct 31 04:45:18.817249 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:18.817193    5831 scope.go:122] "RemoveContainer" containerID="7823764e2d904d4b90a71323a779117a79591823b7c4ac5149f701953a99f72d"
Oct 31 04:45:18.817576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:18.817557    5831 scope.go:122] "RemoveContainer" containerID="72a0ad596bbeb4363ba5f2f83ef5562204aec41c67107a1f13ae3b303377fbdf"
Oct 31 04:45:18.817850 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:45:18.817831    5831 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890365-sttbp_odh-ai-gateway-infra(bd0e19e2-ff75-4dc2-9f42-558dde66113e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e"
Oct 31 04:45:19.931098 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:19.931066    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:20.072086 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.072063    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:20.214958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.214862    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\" (UniqueName: \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\") pod \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") "
Oct 31 04:45:20.214958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.214920    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\") pod \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\" (UID: \"bd0e19e2-ff75-4dc2-9f42-558dde66113e\") "
Oct 31 04:45:20.215459 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.215421    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca" pod "bd0e19e2-ff75-4dc2-9f42-558dde66113e" (UID: "bd0e19e2-ff75-4dc2-9f42-558dde66113e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:20.217114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.217088    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7" pod "bd0e19e2-ff75-4dc2-9f42-558dde66113e" (UID: "bd0e19e2-ff75-4dc2-9f42-558dde66113e"). InnerVolumeSpecName "kube-api-access-qklm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:20.316670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.316636    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklm7\" (UniqueName: \"kubernetes.io/projected/bd0e19e2-ff75-4dc2-9f42-558dde66113e-kube-api-access-qklm7\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:20.316670 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.316665    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd0e19e2-ff75-4dc2-9f42-558dde66113e-maas-api-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:20.831895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.831865    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"
Oct 31 04:45:20.831895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.831877    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp" event={"ID":"bd0e19e2-ff75-4dc2-9f42-558dde66113e","Type":"ContainerDied","Data":"afcb7343470f5d63de2e1cf3a207a4e3c8b8604044cfd5f8b04f78a68fe297bf"}
Oct 31 04:45:20.832186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.831957    5831 scope.go:122] "RemoveContainer" containerID="72a0ad596bbeb4363ba5f2f83ef5562204aec41c67107a1f13ae3b303377fbdf"
Oct 31 04:45:20.984154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:20.984115    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:21.029858 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:21.029826    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890365-sttbp"]
Oct 31 04:45:22.134430 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:22.134398    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" path="/var/lib/kubelet/pods/bd0e19e2-ff75-4dc2-9f42-558dde66113e/volumes"
Oct 31 04:45:26.652234 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:26.652196    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:45:35.560555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:35.556644    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:45:35.560555 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:35.556718    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:45:37.260819 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:37.260786    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:45:37.833214 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:37.833179    5831 scope.go:122] "RemoveContainer" containerID="49df41273e252843403a689c695762e8b989f783a05b6fb430ac88f68ce570b6"
Oct 31 04:45:38.535461 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535424    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"]
Oct 31 04:45:38.535921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535886    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.535921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535899    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535928    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535934    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535948    5831 cpu_manager.go:414] "containerMap: removing container" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.535953    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.536038    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.536048    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.536152 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.536055    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0e19e2-ff75-4dc2-9f42-558dde66113e" containerName="cleanup"
Oct 31 04:45:38.540709 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.540689    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.592047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.592009    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"]
Oct 31 04:45:38.606475 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.606443    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.606680 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.606657    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpfc\" (UniqueName: \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.606735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.606693    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.634777 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.634748    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:45:38.634949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.634779    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:45:38.634949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.634783    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:45:38.634949 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.634825    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:45:38.708094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.708060    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpfc\" (UniqueName: \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.708094 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.708096    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.708351 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.708135    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.708835 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.708804    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.710878 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.710855    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.761345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.761301    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpfc\" (UniqueName: \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\") pod \"maas-api-revoke-sub-3d4156f1-4ddjd\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:38.854325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:38.854239    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:39.220853 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:39.220820    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"]
Oct 31 04:45:39.224287 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:45:39.224253    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ab42d2_2d9e_403e_ba9f_41a669179a6c.slice/crio-d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e WatchSource:0}: Error finding container d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e: Status 404 returned error can't find the container with id d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e
Oct 31 04:45:40.054637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:40.054598    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd" event={"ID":"d7ab42d2-2d9e-403e-ba9f-41a669179a6c","Type":"ContainerStarted","Data":"c0698f0902b7a552ae3eeb4a654129f2b070d3f30a0a3d0ea8ad256c3c31867b"}
Oct 31 04:45:40.055154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:40.054656    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd" event={"ID":"d7ab42d2-2d9e-403e-ba9f-41a669179a6c","Type":"ContainerStarted","Data":"d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e"}
Oct 31 04:45:40.161655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:40.161606    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd" podStartSLOduration=2.161591041 podStartE2EDuration="2.161591041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:40.160771853 +0000 UTC m=+5890.175333560" watchObservedRunningTime="2026-10-31 04:45:40.161591041 +0000 UTC m=+5890.176152748"
Oct 31 04:45:41.061826 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:41.061787    5831 generic.go:356] "Generic (PLEG): container finished" podID="d7ab42d2-2d9e-403e-ba9f-41a669179a6c" containerID="c0698f0902b7a552ae3eeb4a654129f2b070d3f30a0a3d0ea8ad256c3c31867b" exitCode=0
Oct 31 04:45:41.062296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:41.061877    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd" event={"ID":"d7ab42d2-2d9e-403e-ba9f-41a669179a6c","Type":"ContainerDied","Data":"c0698f0902b7a552ae3eeb4a654129f2b070d3f30a0a3d0ea8ad256c3c31867b"}
Oct 31 04:45:42.225595 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.225569    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:42.346080 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.346004    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\" (UniqueName: \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\") pod \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") "
Oct 31 04:45:42.346269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.346080    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\") pod \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") "
Oct 31 04:45:42.346269 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.346123    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\") pod \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\" (UID: \"d7ab42d2-2d9e-403e-ba9f-41a669179a6c\") "
Oct 31 04:45:42.346608 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.346582    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca" pod "d7ab42d2-2d9e-403e-ba9f-41a669179a6c" (UID: "d7ab42d2-2d9e-403e-ba9f-41a669179a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:42.348325 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.348300    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca" pod "d7ab42d2-2d9e-403e-ba9f-41a669179a6c" (UID: "d7ab42d2-2d9e-403e-ba9f-41a669179a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:45:42.348424 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.348380    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc" pod "d7ab42d2-2d9e-403e-ba9f-41a669179a6c" (UID: "d7ab42d2-2d9e-403e-ba9f-41a669179a6c"). InnerVolumeSpecName "kube-api-access-dqpfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:42.447246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.447204    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:42.447246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.447238    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:42.447246 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:42.447253    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqpfc\" (UniqueName: \"kubernetes.io/projected/d7ab42d2-2d9e-403e-ba9f-41a669179a6c-kube-api-access-dqpfc\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:45:42.978726 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:45:42.978682    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:45:43.076428 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:43.076394    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"
Oct 31 04:45:43.076624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:43.076399    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd" event={"ID":"d7ab42d2-2d9e-403e-ba9f-41a669179a6c","Type":"ContainerDied","Data":"d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e"}
Oct 31 04:45:43.076624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:43.076559    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d316eac47140b8276e90a8963d77ac493f10981936f2cf843a6b70963a13482e"
Oct 31 04:45:52.877082 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:45:52.877045    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:46:02.284798 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:02.284762    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:46:10.196153 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:10.196113    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:46:13.074676 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:46:13.074634    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:46:20.206482 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.206448    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"]
Oct 31 04:46:20.207019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.206885    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab42d2-2d9e-403e-ba9f-41a669179a6c" containerName="revoke-keys"
Oct 31 04:46:20.207019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.206899    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab42d2-2d9e-403e-ba9f-41a669179a6c" containerName="revoke-keys"
Oct 31 04:46:20.207019 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.207001    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab42d2-2d9e-403e-ba9f-41a669179a6c" containerName="revoke-keys"
Oct 31 04:46:20.210599 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.210578    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.259197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.259167    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:46:20.300719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.300686    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.300719 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.300725    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwtz\" (UniqueName: \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.300945 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.300845    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.305050 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.305027    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:46:20.305183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.305116    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:46:20.305183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.305163    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:46:20.307861 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.307841    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"]
Oct 31 04:46:20.402001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.401966    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.402215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.402049    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.402215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.402068    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwtz\" (UniqueName: \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.402807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.402780    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.404607 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.404589    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.456102 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.456070    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwtz\" (UniqueName: \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\") pod \"maas-api-revoke-sub-e2fc9c5b-kx6sv\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.523523 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.523441    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:20.899629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:20.899508    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"]
Oct 31 04:46:20.902894 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:46:20.902865    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91484e9_bb28_44ef_a9ff_1efb1e3b19bf.slice/crio-6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33 WatchSource:0}: Error finding container 6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33: Status 404 returned error can't find the container with id 6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33
Oct 31 04:46:21.352199 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:21.352159    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv" event={"ID":"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf","Type":"ContainerStarted","Data":"96cd2e87b8dc11c77c2782d44c336481fe19e2a3abeacec32602a819f73aa5eb"}
Oct 31 04:46:21.352643 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:21.352246    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv" event={"ID":"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf","Type":"ContainerStarted","Data":"6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33"}
Oct 31 04:46:21.457892 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:21.457789    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv" podStartSLOduration=1.45777467 podStartE2EDuration="1.45777467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:21.456842758 +0000 UTC m=+5931.471404457" watchObservedRunningTime="2026-10-31 04:46:21.45777467 +0000 UTC m=+5931.472336377"
Oct 31 04:46:22.359958 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:22.359922    5831 generic.go:356] "Generic (PLEG): container finished" podID="d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" containerID="96cd2e87b8dc11c77c2782d44c336481fe19e2a3abeacec32602a819f73aa5eb" exitCode=0
Oct 31 04:46:22.360360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:22.359982    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv" event={"ID":"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf","Type":"ContainerDied","Data":"96cd2e87b8dc11c77c2782d44c336481fe19e2a3abeacec32602a819f73aa5eb"}
Oct 31 04:46:23.511035 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.511009    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:23.635445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.635361    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\") pod \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") "
Oct 31 04:46:23.635637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.635442    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\") pod \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") "
Oct 31 04:46:23.635637 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.635502    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\" (UniqueName: \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\") pod \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\" (UID: \"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf\") "
Oct 31 04:46:23.636197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.636167    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca" pod "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" (UID: "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:46:23.640706 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.640669    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz" pod "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" (UID: "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf"). InnerVolumeSpecName "kube-api-access-fvwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:46:23.642653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.642625    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca" pod "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" (UID: "d91484e9-bb28-44ef-a9ff-1efb1e3b19bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:46:23.736447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.736406    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:23.736447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.736443    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:23.736447 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:23.736456    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwtz\" (UniqueName: \"kubernetes.io/projected/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf-kube-api-access-fvwtz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:24.373568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:24.373515    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"
Oct 31 04:46:24.373568 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:24.373544    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv" event={"ID":"d91484e9-bb28-44ef-a9ff-1efb1e3b19bf","Type":"ContainerDied","Data":"6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33"}
Oct 31 04:46:24.373799 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:24.373596    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d9d840fd368c6b1c9be194c21e31f03d098bfb56007118e19bb8d64a8c3bd33"
Oct 31 04:46:30.914170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.914085    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"]
Oct 31 04:46:30.914653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.914634    5831 cpu_manager.go:414] "containerMap: removing container" podUID="d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" containerName="revoke-keys"
Oct 31 04:46:30.914733 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.914656    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" containerName="revoke-keys"
Oct 31 04:46:30.914837 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.914820    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" containerName="revoke-keys"
Oct 31 04:46:30.918393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.918372    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:30.970416 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:30.970369    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"]
Oct 31 04:46:31.011857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.011814    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.012061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.011881    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7nj\" (UniqueName: \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.012061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.011950    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.012773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.012746    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:46:31.012773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.012761    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:46:31.012981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.012794    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:46:31.012981 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.012755    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:46:31.112751 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.112720    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.112932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.112784    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7nj\" (UniqueName: \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.112932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.112812    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.113398 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.113378    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.115347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.115313    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.166474 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.166401    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7nj\" (UniqueName: \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\") pod \"maas-api-revoke-sub-5a29efb3-ld9mp\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.232346 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.232316    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:31.606158 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:31.606105    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"]
Oct 31 04:46:31.608084 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:46:31.608056    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2b1c32_d02a_4b3d_9fe2_4e37a3cc138f.slice/crio-da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71 WatchSource:0}: Error finding container da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71: Status 404 returned error can't find the container with id da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71
Oct 31 04:46:32.427376 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:32.427338    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp" event={"ID":"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f","Type":"ContainerStarted","Data":"337627438ff607d711332176b3abf302f780d70c113e4c9173a426312510081c"}
Oct 31 04:46:32.427845 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:32.427401    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp" event={"ID":"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f","Type":"ContainerStarted","Data":"da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71"}
Oct 31 04:46:32.532489 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:32.532437    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp" podStartSLOduration=2.532423513 podStartE2EDuration="2.532423513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:32.532286156 +0000 UTC m=+5942.546847864" watchObservedRunningTime="2026-10-31 04:46:32.532423513 +0000 UTC m=+5942.546985220"
Oct 31 04:46:32.880474 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:32.880428    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:46:33.440653 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:33.440599    5831 generic.go:356] "Generic (PLEG): container finished" podID="7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" containerID="337627438ff607d711332176b3abf302f780d70c113e4c9173a426312510081c" exitCode=0
Oct 31 04:46:33.441177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:33.440738    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp" event={"ID":"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f","Type":"ContainerDied","Data":"337627438ff607d711332176b3abf302f780d70c113e4c9173a426312510081c"}
Oct 31 04:46:35.231769 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.226691    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"]
Oct 31 04:46:35.263556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.237652    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-6z6vz"]
Oct 31 04:46:35.346073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.345742    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:35.486215 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.486164    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\" (UniqueName: \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\") pod \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") "
Oct 31 04:46:35.486961 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.486936    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\") pod \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") "
Oct 31 04:46:35.487862 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.487843    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\") pod \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\" (UID: \"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f\") "
Oct 31 04:46:35.488736 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.487548    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca" pod "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" (UID: "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:46:35.518625 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.518587    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp" event={"ID":"7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f","Type":"ContainerDied","Data":"da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71"}
Oct 31 04:46:35.518946 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.518909    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da75bf6f425f47a9c2d719e20cb61de9767dfbf45e4f662d589c263d7ab9fb71"
Oct 31 04:46:35.519170 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.519154    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"
Oct 31 04:46:35.521825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.521779    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj" pod "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" (UID: "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f"). InnerVolumeSpecName "kube-api-access-qh7nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:46:35.524678 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.524640    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca" pod "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" (UID: "7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:46:35.555627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.555097    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:46:35.555627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.555159    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:46:35.590305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.590209    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:35.590305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.590253    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:35.590305 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:35.590273    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7nj\" (UniqueName: \"kubernetes.io/projected/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f-kube-api-access-qh7nj\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:36.164277 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:36.164228    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0699f9f-1c17-4abe-a032-45b31326adc4" path="/var/lib/kubelet/pods/f0699f9f-1c17-4abe-a032-45b31326adc4/volumes"
Oct 31 04:46:42.980806 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:46:42.980760    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:46:43.579785 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:43.579753    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:46:48.141484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.141442    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"]
Oct 31 04:46:48.187445 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.187415    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ddjd"]
Oct 31 04:46:48.253679 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.253642    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"]
Oct 31 04:46:48.254096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.254082    5831 cpu_manager.go:414] "containerMap: removing container" podUID="7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" containerName="revoke-keys"
Oct 31 04:46:48.254143 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.254099    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" containerName="revoke-keys"
Oct 31 04:46:48.254212 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.254203    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" containerName="revoke-keys"
Oct 31 04:46:48.257821 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.257800    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.306549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.306498    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:46:48.309495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.309453    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:46:48.309934 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.309842    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:46:48.312793 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.312760    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"]
Oct 31 04:46:48.341734 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.341684    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsj6h\" (UniqueName: \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.341917 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.341899    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.341964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.341932    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.406818 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.406730    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:46:48.442807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.442777    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.443000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.442813    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.443000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.442840    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsj6h\" (UniqueName: \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.443426 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.443404    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.445338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.445314    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.495588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.495521    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsj6h\" (UniqueName: \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\") pod \"maas-api-revoke-sub-3d4156f1-7tn9x\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.571031 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.570997    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:48.948735 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:48.948697    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"]
Oct 31 04:46:48.950178 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:46:48.950135    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d2b4da_c315_440e_a228_a30d99f53e1a.slice/crio-bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653 WatchSource:0}: Error finding container bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653: Status 404 returned error can't find the container with id bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653
Oct 31 04:46:49.611880 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:49.611787    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x" event={"ID":"09d2b4da-c315-440e-a228-a30d99f53e1a","Type":"ContainerStarted","Data":"d72b157ef0adfb14fed2928c09d1bfe3dbab72abdec7f9e2cce5aadb90f9b3b5"}
Oct 31 04:46:49.612271 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:49.611889    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x" event={"ID":"09d2b4da-c315-440e-a228-a30d99f53e1a","Type":"ContainerStarted","Data":"bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653"}
Oct 31 04:46:49.717255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:49.717197    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x" podStartSLOduration=1.717183644 podStartE2EDuration="1.717183644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:49.71670952 +0000 UTC m=+5959.731271227" watchObservedRunningTime="2026-10-31 04:46:49.717183644 +0000 UTC m=+5959.731745349"
Oct 31 04:46:50.134723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:50.134687    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ab42d2-2d9e-403e-ba9f-41a669179a6c" path="/var/lib/kubelet/pods/d7ab42d2-2d9e-403e-ba9f-41a669179a6c/volumes"
Oct 31 04:46:50.619627 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:50.619588    5831 generic.go:356] "Generic (PLEG): container finished" podID="09d2b4da-c315-440e-a228-a30d99f53e1a" containerID="d72b157ef0adfb14fed2928c09d1bfe3dbab72abdec7f9e2cce5aadb90f9b3b5" exitCode=0
Oct 31 04:46:50.620068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:50.619686    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x" event={"ID":"09d2b4da-c315-440e-a228-a30d99f53e1a","Type":"ContainerDied","Data":"d72b157ef0adfb14fed2928c09d1bfe3dbab72abdec7f9e2cce5aadb90f9b3b5"}
Oct 31 04:46:51.772484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.772455    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:51.886020 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.885936    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\") pod \"09d2b4da-c315-440e-a228-a30d99f53e1a\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") "
Oct 31 04:46:51.886020 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.886006    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\") pod \"09d2b4da-c315-440e-a228-a30d99f53e1a\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") "
Oct 31 04:46:51.886238 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.886160    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\" (UniqueName: \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\") pod \"09d2b4da-c315-440e-a228-a30d99f53e1a\" (UID: \"09d2b4da-c315-440e-a228-a30d99f53e1a\") "
Oct 31 04:46:51.886466 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.886440    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca" pod "09d2b4da-c315-440e-a228-a30d99f53e1a" (UID: "09d2b4da-c315-440e-a228-a30d99f53e1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:46:51.888509 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.888486    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h" pod "09d2b4da-c315-440e-a228-a30d99f53e1a" (UID: "09d2b4da-c315-440e-a228-a30d99f53e1a"). InnerVolumeSpecName "kube-api-access-rsj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:46:51.888889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.888855    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca" pod "09d2b4da-c315-440e-a228-a30d99f53e1a" (UID: "09d2b4da-c315-440e-a228-a30d99f53e1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:46:51.987150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.987120    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsj6h\" (UniqueName: \"kubernetes.io/projected/09d2b4da-c315-440e-a228-a30d99f53e1a-kube-api-access-rsj6h\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:51.987150 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.987148    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:51.987367 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:51.987159    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d2b4da-c315-440e-a228-a30d99f53e1a-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:46:52.633593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:52.633558    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x" event={"ID":"09d2b4da-c315-440e-a228-a30d99f53e1a","Type":"ContainerDied","Data":"bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653"}
Oct 31 04:46:52.633593 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:52.633592    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"
Oct 31 04:46:52.633842 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:46:52.633614    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bccffe0352cea0d0478c5cae6b8a2c0a0f43a9e17f4cdcee3500939b4b34f653"
Oct 31 04:47:02.571873 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:02.571835    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:47:04.843350 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.843309    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:47:04.853251 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.853212    5831 cpu_manager.go:414] "containerMap: removing container" podUID="09d2b4da-c315-440e-a228-a30d99f53e1a" containerName="revoke-keys"
Oct 31 04:47:04.853487 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.853469    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d2b4da-c315-440e-a228-a30d99f53e1a" containerName="revoke-keys"
Oct 31 04:47:04.853822 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.853801    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d2b4da-c315-440e-a228-a30d99f53e1a" containerName="revoke-keys"
Oct 31 04:47:04.881802 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.881773    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.034134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:04.981231    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.034613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.034573    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.036654 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.036593    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjh7w\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.037060 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.037041    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.037384 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.037307    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.038051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.038020    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.038455 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.038381    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.038671 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.038654    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.039067 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.039006    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.104253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.013686    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-cc1991-openshift-default-dockercfg-zr44t\""
Oct 31 04:47:05.104253 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.104094    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:47:05.154557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.141238    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.167219 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.142006    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.167612 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.167578    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.167918 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.167799    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168117 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168103    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168291    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjh7w\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168441 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168428    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168662 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168646    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168815 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168802    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.168995 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168983    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.193567 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.169289    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.193857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.184391    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.194009 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.168699    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.200630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.197105    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.203059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.203028    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.207919 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.207892    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.231925 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.231847    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjh7w\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.237906 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.237856    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\") pod \"e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") " pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.284177 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.283899    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:05.713650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.713441    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:47:05.716160 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:47:05.716134    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d03b45_79bb_4b18_88d8_471da8dd66de.slice/crio-a0feb5c4cd022ef5c6512d9d2ec890726991ae6d600fb26eee12d173ec776e57 WatchSource:0}: Error finding container a0feb5c4cd022ef5c6512d9d2ec890726991ae6d600fb26eee12d173ec776e57: Status 404 returned error can't find the container with id a0feb5c4cd022ef5c6512d9d2ec890726991ae6d600fb26eee12d173ec776e57
Oct 31 04:47:05.718339 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.718308    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:05.718400 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.718375    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:05.718443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.718401    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:05.762803 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:05.762774    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerStarted","Data":"a0feb5c4cd022ef5c6512d9d2ec890726991ae6d600fb26eee12d173ec776e57"}
Oct 31 04:47:06.770888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:06.770847    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerStarted","Data":"9f643ddc39d934b5779c68a53d3cc1549b317f411dfa60e9def55d702a13cfb8"}
Oct 31 04:47:06.927092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:06.927037    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podStartSLOduration=2.927018368 podStartE2EDuration="2.927018368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:06.926189099 +0000 UTC m=+5976.940750789" watchObservedRunningTime="2026-10-31 04:47:06.927018368 +0000 UTC m=+5976.941580073"
Oct 31 04:47:07.042204 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.042120    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"]
Oct 31 04:47:07.045005 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.044969    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-c77dl"]
Oct 31 04:47:07.284434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.284407    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:07.289889 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.289863    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:07.778104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.778068    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:07.779021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:07.779002    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:08.141138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:08.141049    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac483155-efef-4681-9f5d-2224eb31833d" path="/var/lib/kubelet/pods/ac483155-efef-4681-9f5d-2224eb31833d/volumes"
Oct 31 04:47:12.998316 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:47:12.998280    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:47:15.787738 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:15.787698    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:47:27.854897 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:27.854859    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:47:31.955558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.955495    5831 generic.go:356] "Generic (PLEG): container finished" podID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerID="9f643ddc39d934b5779c68a53d3cc1549b317f411dfa60e9def55d702a13cfb8" exitCode=0
Oct 31 04:47:31.956147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.955574    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerDied","Data":"9f643ddc39d934b5779c68a53d3cc1549b317f411dfa60e9def55d702a13cfb8"}
Oct 31 04:47:31.956147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.956100    5831 scope.go:122] "RemoveContainer" containerID="9f643ddc39d934b5779c68a53d3cc1549b317f411dfa60e9def55d702a13cfb8"
Oct 31 04:47:31.957411 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.957384    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:31.957483 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.957452    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:31.957539 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:31.957485    5831 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 04:47:32.054613 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:32.054577    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"]
Oct 31 04:47:32.058440 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:32.058402    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-w6gqf"]
Oct 31 04:47:32.135788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:32.135755    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ecba983-4cb9-4b84-8d89-418a8ca2e616" path="/var/lib/kubelet/pods/8ecba983-4cb9-4b84-8d89-418a8ca2e616/volumes"
Oct 31 04:47:32.964280 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:32.964245    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerStarted","Data":"eb71d621439a980b5c8360c8f31b0f02f04951ae4bb8ef680eef4786a6f80653"}
Oct 31 04:47:33.294333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:33.294168    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:33.294674 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:33.294651    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body=
Oct 31 04:47:33.294857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:33.294828    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused"
Oct 31 04:47:34.295556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:34.284858    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body=
Oct 31 04:47:34.295556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:34.284917    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused"
Oct 31 04:47:35.284788 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:35.284749    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body=
Oct 31 04:47:35.285083 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:35.285055    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused"
Oct 31 04:47:36.290564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:36.285903    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body=
Oct 31 04:47:36.290564 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:36.285965    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused"
Oct 31 04:47:37.290964 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:37.290923    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body=
Oct 31 04:47:37.291630 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:37.291598    5831 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused"
Oct 31 04:47:37.778064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:37.778029    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:37.871335 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:37.871302    5831 scope.go:122] "RemoveContainer" containerID="32661fa6b9f145472123f14e1b8521f2d4bf813e953d3c4e675de4a24517257b"
Oct 31 04:47:38.290151 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.290123    5831 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:38.291258 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.291234    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:47:38.921686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.921658    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:47:38.932282 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.932257    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:47:38.932480 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.932325    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:47:38.936088 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.936069    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:47:38.942015 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.941996    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:47:38.945327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:38.945312    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:47:42.892276 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:42.892231    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:47:42.981026 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:47:42.980992    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:47:45.043329 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:45.043295    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"]
Oct 31 04:47:45.089330 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:45.089293    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-498qh"]
Oct 31 04:47:46.134168 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:47:46.134132    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4a6bde-923b-48bb-adec-5954b57daeba" path="/var/lib/kubelet/pods/6e4a6bde-923b-48bb-adec-5954b57daeba/volumes"
Oct 31 04:48:08.175665 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:08.175579    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:48:12.977880 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:48:12.977840    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:48:22.042524 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.042481    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"]
Oct 31 04:48:22.088322 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.088289    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m75df"]
Oct 31 04:48:22.136176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.136135    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5977949b-9cf9-48aa-89c3-13ff09a9551c" path="/var/lib/kubelet/pods/5977949b-9cf9-48aa-89c3-13ff09a9551c/volumes"
Oct 31 04:48:22.495492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.495452    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"]
Oct 31 04:48:22.499611 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.499587    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.548775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.548626    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:48:22.548775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.548712    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:48:22.594690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.594652    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:48:22.594690 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.594650    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:48:22.597578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.597558    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"]
Oct 31 04:48:22.642168 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.642134    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.642339 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.642190    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.642339 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.642278    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458g2\" (UniqueName: \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.743796 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.743762    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.743999 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.743816    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.743999 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.743858    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458g2\" (UniqueName: \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.744484 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.744456    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.746620 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.746566    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.797288 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.797261    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458g2\" (UniqueName: \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\") pod \"maas-api-revoke-sub-06e39e43-ww29p\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:22.813084 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:22.813054    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:23.185300 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:23.185268    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"]
Oct 31 04:48:23.189256 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:48:23.189216    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7aeb125_6aec_4ae8_b671_ac822519a42d.slice/crio-a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94 WatchSource:0}: Error finding container a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94: Status 404 returned error can't find the container with id a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94
Oct 31 04:48:23.392046 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:23.392015    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p" event={"ID":"e7aeb125-6aec-4ae8-b671-ac822519a42d","Type":"ContainerStarted","Data":"a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94"}
Oct 31 04:48:24.400606 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:24.400486    5831 generic.go:356] "Generic (PLEG): container finished" podID="e7aeb125-6aec-4ae8-b671-ac822519a42d" containerID="b31d5fb09e51a5b840543809ad870b2346ea2039c84dc0dc950ca26c1b147a7d" exitCode=0
Oct 31 04:48:24.400606 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:24.400587    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p" event={"ID":"e7aeb125-6aec-4ae8-b671-ac822519a42d","Type":"ContainerDied","Data":"b31d5fb09e51a5b840543809ad870b2346ea2039c84dc0dc950ca26c1b147a7d"}
Oct 31 04:48:25.556492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.556471    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:25.670142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.670044    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\" (UniqueName: \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\") pod \"e7aeb125-6aec-4ae8-b671-ac822519a42d\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") "
Oct 31 04:48:25.670321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.670170    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\") pod \"e7aeb125-6aec-4ae8-b671-ac822519a42d\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") "
Oct 31 04:48:25.670321 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.670235    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\") pod \"e7aeb125-6aec-4ae8-b671-ac822519a42d\" (UID: \"e7aeb125-6aec-4ae8-b671-ac822519a42d\") "
Oct 31 04:48:25.670559 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.670503    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca" pod "e7aeb125-6aec-4ae8-b671-ac822519a42d" (UID: "e7aeb125-6aec-4ae8-b671-ac822519a42d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:25.672932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.672906    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2" pod "e7aeb125-6aec-4ae8-b671-ac822519a42d" (UID: "e7aeb125-6aec-4ae8-b671-ac822519a42d"). InnerVolumeSpecName "kube-api-access-458g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:25.679980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.679953    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca" pod "e7aeb125-6aec-4ae8-b671-ac822519a42d" (UID: "e7aeb125-6aec-4ae8-b671-ac822519a42d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:48:25.771697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.771659    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:25.771697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.771689    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458g2\" (UniqueName: \"kubernetes.io/projected/e7aeb125-6aec-4ae8-b671-ac822519a42d-kube-api-access-458g2\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:25.771697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:25.771702    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7aeb125-6aec-4ae8-b671-ac822519a42d-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:26.414043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:26.413959    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"
Oct 31 04:48:26.414043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:26.413984    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p" event={"ID":"e7aeb125-6aec-4ae8-b671-ac822519a42d","Type":"ContainerDied","Data":"a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94"}
Oct 31 04:48:26.414043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:26.414036    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2940c0c6598468a34ced1b2d4e22d992def5a0d4bc3ac6b35e7ca538ef7bc94"
Oct 31 04:48:33.044298 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:33.044261    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"]
Oct 31 04:48:33.090193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:33.090163    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bdhf8"]
Oct 31 04:48:34.134991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:34.134959    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef3081a-7086-4d1b-9a65-eb364f5d57cc" path="/var/lib/kubelet/pods/1ef3081a-7086-4d1b-9a65-eb364f5d57cc/volumes"
Oct 31 04:48:37.291911 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:37.291868    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:48:37.908405 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:37.908373    5831 scope.go:122] "RemoveContainer" containerID="d3be08ec4df4c4a4c44a3231905d997c15e1146112db96bb2c02b2fdbe8b005e"
Oct 31 04:48:37.920203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:37.920175    5831 scope.go:122] "RemoveContainer" containerID="6aab540843d92cd315f67d1e260f3c9d2d107f833fe2d1c63285122ecc66b887"
Oct 31 04:48:42.977666 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:48:42.977621    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:48:50.362488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:48:50.362446    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:49:07.067059 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:07.067023    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"]
Oct 31 04:49:07.112914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:07.112878    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rswvl"]
Oct 31 04:49:08.135138 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:08.135105    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb735ec5-6c3c-42a1-987a-357b4c91655d" path="/var/lib/kubelet/pods/cb735ec5-6c3c-42a1-987a-357b4c91655d/volumes"
Oct 31 04:49:12.978324 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:49:12.978277    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:49:19.671766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:19.671735    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:49:19.672237 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:19.671985    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" containerID="cri-o://eb71d621439a980b5c8360c8f31b0f02f04951ae4bb8ef680eef4786a6f80653" gracePeriod=30
Oct 31 04:49:21.222825 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.222784    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"]
Oct 31 04:49:21.269797 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.269755    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7tn9x"]
Oct 31 04:49:21.334824 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.334788    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"]
Oct 31 04:49:21.335338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.335322    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e7aeb125-6aec-4ae8-b671-ac822519a42d" containerName="revoke-keys"
Oct 31 04:49:21.335393 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.335340    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7aeb125-6aec-4ae8-b671-ac822519a42d" containerName="revoke-keys"
Oct 31 04:49:21.335454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.335444    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7aeb125-6aec-4ae8-b671-ac822519a42d" containerName="revoke-keys"
Oct 31 04:49:21.338879 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.338858    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.390230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.390197    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:49:21.390442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.390348    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:49:21.390442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.390410    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:49:21.390618 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.390473    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:49:21.393308 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.393279    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"]
Oct 31 04:49:21.437864 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.437830    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.438047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.437907    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkz8j\" (UniqueName: \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.438047 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.437970    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.539221 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.539127    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.539414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.539243    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.539414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.539260    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkz8j\" (UniqueName: \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.539948 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.539925    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.541730 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.541701    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.592201 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.592159    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkz8j\" (UniqueName: \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\") pod \"maas-api-revoke-sub-3d4156f1-g2ghb\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:21.655888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:21.655851    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:22.026766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:22.026740    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"]
Oct 31 04:49:22.029152 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:49:22.029121    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36369f2_8c5b_40d3_b6c3_806ac47cf65f.slice/crio-84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f WatchSource:0}: Error finding container 84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f: Status 404 returned error can't find the container with id 84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f
Oct 31 04:49:22.134048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:22.134019    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d2b4da-c315-440e-a228-a30d99f53e1a" path="/var/lib/kubelet/pods/09d2b4da-c315-440e-a228-a30d99f53e1a/volumes"
Oct 31 04:49:22.809414 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:22.809371    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb" event={"ID":"e36369f2-8c5b-40d3-b6c3-806ac47cf65f","Type":"ContainerStarted","Data":"e4d4dd2c7a54eaf13793c56c2d8d368251a67207e29d40654bc525b6344f2d1e"}
Oct 31 04:49:22.809999 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:22.809438    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb" event={"ID":"e36369f2-8c5b-40d3-b6c3-806ac47cf65f","Type":"ContainerStarted","Data":"84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f"}
Oct 31 04:49:22.916928 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:22.916876    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb" podStartSLOduration=1.916860809 podStartE2EDuration="1.916860809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:22.915271111 +0000 UTC m=+6112.929832818" watchObservedRunningTime="2026-10-31 04:49:22.916860809 +0000 UTC m=+6112.931422517"
Oct 31 04:49:23.293345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:23.293310    5831 patch_prober.go:29] interesting pod/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:49:23.293525 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:23.293367    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:49:23.818917 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:23.818871    5831 generic.go:356] "Generic (PLEG): container finished" podID="e36369f2-8c5b-40d3-b6c3-806ac47cf65f" containerID="e4d4dd2c7a54eaf13793c56c2d8d368251a67207e29d40654bc525b6344f2d1e" exitCode=0
Oct 31 04:49:23.819382 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:23.818941    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb" event={"ID":"e36369f2-8c5b-40d3-b6c3-806ac47cf65f","Type":"ContainerDied","Data":"e4d4dd2c7a54eaf13793c56c2d8d368251a67207e29d40654bc525b6344f2d1e"}
Oct 31 04:49:24.827932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.827899    5831 generic.go:356] "Generic (PLEG): container finished" podID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerID="eb71d621439a980b5c8360c8f31b0f02f04951ae4bb8ef680eef4786a6f80653" exitCode=0
Oct 31 04:49:24.828381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.827975    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerDied","Data":"eb71d621439a980b5c8360c8f31b0f02f04951ae4bb8ef680eef4786a6f80653"}
Oct 31 04:49:24.828381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.828054    5831 scope.go:122] "RemoveContainer" containerID="9f643ddc39d934b5779c68a53d3cc1549b317f411dfa60e9def55d702a13cfb8"
Oct 31 04:49:24.959027 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.958955    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:49:24.969685 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969660    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.969844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969694    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.969844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969739    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.969844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969794    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.969844 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969834    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.970051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969861    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.970051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969888    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.970051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969912    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.970051 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.969945    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\") pod \"b3d03b45-79bb-4b18-88d8-471da8dd66de\" (UID: \"b3d03b45-79bb-4b18-88d8-471da8dd66de\") "
Oct 31 04:49:24.970275 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970180    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:24.970360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970308    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:24.970360 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970341    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:24.970494 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970351    5831 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:24.970579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970473    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:24.970579 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.970566    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:24.972544 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.972495    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:24.972664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.972560    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "kube-api-access-sjh7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:24.972664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.972608    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:24.972664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.972617    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo" pod "b3d03b45-79bb-4b18-88d8-471da8dd66de" (UID: "b3d03b45-79bb-4b18-88d8-471da8dd66de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:49:24.996176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:24.996153    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:25.071073 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071041    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\") pod \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") "
Oct 31 04:49:25.071259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071132    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\") pod \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") "
Oct 31 04:49:25.071259 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071155    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\" (UniqueName: \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\") pod \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\" (UID: \"e36369f2-8c5b-40d3-b6c3-806ac47cf65f\") "
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071325    5831 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-token\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071337    5831 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-podinfo\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071346    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjh7w\" (UniqueName: \"kubernetes.io/projected/b3d03b45-79bb-4b18-88d8-471da8dd66de-kube-api-access-sjh7w\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071378    5831 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-envoy\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071389    5831 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-credential-socket\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071398    5831 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-workload-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071403 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071406    5831 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d03b45-79bb-4b18-88d8-471da8dd66de-istio-data\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071415    5831 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d03b45-79bb-4b18-88d8-471da8dd66de-istiod-ca-cert\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.071761 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.071470    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca" pod "e36369f2-8c5b-40d3-b6c3-806ac47cf65f" (UID: "e36369f2-8c5b-40d3-b6c3-806ac47cf65f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:25.073356 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.073334    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca" pod "e36369f2-8c5b-40d3-b6c3-806ac47cf65f" (UID: "e36369f2-8c5b-40d3-b6c3-806ac47cf65f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:49:25.073477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.073459    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j" pod "e36369f2-8c5b-40d3-b6c3-806ac47cf65f" (UID: "e36369f2-8c5b-40d3-b6c3-806ac47cf65f"). InnerVolumeSpecName "kube-api-access-bkz8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:25.172661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.172630    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.172661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.172658    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.172661 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.172669    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkz8j\" (UniqueName: \"kubernetes.io/projected/e36369f2-8c5b-40d3-b6c3-806ac47cf65f-kube-api-access-bkz8j\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:25.836333 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.836304    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"
Oct 31 04:49:25.836805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.836348    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv" event={"ID":"b3d03b45-79bb-4b18-88d8-471da8dd66de","Type":"ContainerDied","Data":"a0feb5c4cd022ef5c6512d9d2ec890726991ae6d600fb26eee12d173ec776e57"}
Oct 31 04:49:25.836805 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.836433    5831 scope.go:122] "RemoveContainer" containerID="eb71d621439a980b5c8360c8f31b0f02f04951ae4bb8ef680eef4786a6f80653"
Oct 31 04:49:25.837807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.837713    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"
Oct 31 04:49:25.837807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.837741    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb" event={"ID":"e36369f2-8c5b-40d3-b6c3-806ac47cf65f","Type":"ContainerDied","Data":"84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f"}
Oct 31 04:49:25.837807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:25.837803    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84873c06591c809721885815b849d66cb8e7c3cd7f2e8431576db9e379d7425f"
Oct 31 04:49:26.087731 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:26.087629    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:49:26.091890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:26.091864    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cc1991-openshift-default-546d4d6df7kw9nv"]
Oct 31 04:49:26.135255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:26.135224    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" path="/var/lib/kubelet/pods/b3d03b45-79bb-4b18-88d8-471da8dd66de/volumes"
Oct 31 04:49:27.120911 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:27.120874    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"]
Oct 31 04:49:27.123176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:27.123149    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-khstn"]
Oct 31 04:49:28.134354 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:28.134320    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b43724e-e4b5-4f8d-a543-d5fee0f09fbc" path="/var/lib/kubelet/pods/5b43724e-e4b5-4f8d-a543-d5fee0f09fbc/volumes"
Oct 31 04:49:35.768784 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:35.768754    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:49:37.958061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:37.958018    5831 scope.go:122] "RemoveContainer" containerID="973fa66222a3c379b041dd4dd8f362021f3c63e4588f1188dbff757ca09c4c18"
Oct 31 04:49:37.990194 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:37.990166    5831 scope.go:122] "RemoveContainer" containerID="730ccb855cc9078a18b8c84338a84e033fb263f12ea2738d48ac26f049439fc4"
Oct 31 04:49:38.044997 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:38.044968    5831 scope.go:122] "RemoveContainer" containerID="e10c73a1e74680c3896ca4cd72296a64cb5182b4c92219a9d86c6c49769344d1"
Oct 31 04:49:40.124120 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:40.124076    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 04:49:40.124692 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:40.124316    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" podUID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" containerName="manager" containerID="cri-o://ecb0e550d411f360efc20a0d4c26756b35b6e8125224557a7dba7f092b769b0b" gracePeriod=10
Oct 31 04:49:40.977673 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:40.977642    5831 generic.go:356] "Generic (PLEG): container finished" podID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" containerID="ecb0e550d411f360efc20a0d4c26756b35b6e8125224557a7dba7f092b769b0b" exitCode=0
Oct 31 04:49:40.977866 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:40.977725    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" event={"ID":"3778b1c5-f1c0-43d6-b45c-c121a45904d2","Type":"ContainerDied","Data":"ecb0e550d411f360efc20a0d4c26756b35b6e8125224557a7dba7f092b769b0b"}
Oct 31 04:49:41.091712 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.091686    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 04:49:41.202651 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.202623    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\") pod \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") "
Oct 31 04:49:41.203119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.202760    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\" (UniqueName: \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\") pod \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\" (UID: \"3778b1c5-f1c0-43d6-b45c-c121a45904d2\") "
Oct 31 04:49:41.203119 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.203040    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume" pod "3778b1c5-f1c0-43d6-b45c-c121a45904d2" (UID: "3778b1c5-f1c0-43d6-b45c-c121a45904d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:49:41.205366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.205343    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4" pod "3778b1c5-f1c0-43d6-b45c-c121a45904d2" (UID: "3778b1c5-f1c0-43d6-b45c-c121a45904d2"). InnerVolumeSpecName "kube-api-access-l7cj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:41.303718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.303685    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cj4\" (UniqueName: \"kubernetes.io/projected/3778b1c5-f1c0-43d6-b45c-c121a45904d2-kube-api-access-l7cj4\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:41.303718 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.303711    5831 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3778b1c5-f1c0-43d6-b45c-c121a45904d2-extensions-socket-volume\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:41.985442 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.985408    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc" event={"ID":"3778b1c5-f1c0-43d6-b45c-c121a45904d2","Type":"ContainerDied","Data":"db4e2627b788600e71f8b6eac69a81bae83dc76e1c13c9038f56dbc25a351029"}
Oct 31 04:49:41.985659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.985447    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"
Oct 31 04:49:41.985659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:41.985478    5831 scope.go:122] "RemoveContainer" containerID="ecb0e550d411f360efc20a0d4c26756b35b6e8125224557a7dba7f092b769b0b"
Oct 31 04:49:42.185345 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:42.185312    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 04:49:42.231888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:42.231852    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ss2vc"]
Oct 31 04:49:42.977233 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:49:42.977195    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:49:44.134147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:44.134112    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" path="/var/lib/kubelet/pods/3778b1c5-f1c0-43d6-b45c-c121a45904d2/volumes"
Oct 31 04:49:47.042707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:47.042675    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"]
Oct 31 04:49:47.088594 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:47.088557    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gvp66"]
Oct 31 04:49:48.134169 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:49:48.134133    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add09104-ac1c-4a52-a325-23a954e30eed" path="/var/lib/kubelet/pods/add09104-ac1c-4a52-a325-23a954e30eed/volumes"
Oct 31 04:50:12.978190 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:50:12.978148    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:50:38.306014 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:38.305968    5831 scope.go:122] "RemoveContainer" containerID="94cd423013d9cfa631a4315ba6815e5d06f7c2e59dcb377cdeea214ecfefc9c2"
Oct 31 04:50:38.316090 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:38.316059    5831 scope.go:122] "RemoveContainer" containerID="098da48024be8a22b160830ddd8694b641929ed1b3d2d4c9a2b84b653ea41b68"
Oct 31 04:50:42.979315 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:50:42.979277    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:50:43.719071 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719036    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"]
Oct 31 04:50:43.719466 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719454    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e36369f2-8c5b-40d3-b6c3-806ac47cf65f" containerName="revoke-keys"
Oct 31 04:50:43.719517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719468    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36369f2-8c5b-40d3-b6c3-806ac47cf65f" containerName="revoke-keys"
Oct 31 04:50:43.719517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719481    5831 cpu_manager.go:414] "containerMap: removing container" podUID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" containerName="manager"
Oct 31 04:50:43.719517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719487    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" containerName="manager"
Oct 31 04:50:43.719517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719514    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719520    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719559    5831 cpu_manager.go:414] "containerMap: removing container" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719568    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719641    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719651    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36369f2-8c5b-40d3-b6c3-806ac47cf65f" containerName="revoke-keys"
Oct 31 04:50:43.719701 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.719658    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="3778b1c5-f1c0-43d6-b45c-c121a45904d2" containerName="manager"
Oct 31 04:50:43.722914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.722899    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.769849 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.769819    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"]
Oct 31 04:50:43.812813 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.812783    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c6vvr\""
Oct 31 04:50:43.813029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.813004    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j62v\" (UniqueName: \"kubernetes.io/projected/81e4568f-757b-469b-9f75-77565cc0ae98-kube-api-access-9j62v\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.813104 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.813088    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81e4568f-757b-469b-9f75-77565cc0ae98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.913886 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.913847    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81e4568f-757b-469b-9f75-77565cc0ae98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.914087 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.913976    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j62v\" (UniqueName: \"kubernetes.io/projected/81e4568f-757b-469b-9f75-77565cc0ae98-kube-api-access-9j62v\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.914306 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.914285    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81e4568f-757b-469b-9f75-77565cc0ae98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:43.967787 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:43.967758    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j62v\" (UniqueName: \"kubernetes.io/projected/81e4568f-757b-469b-9f75-77565cc0ae98-kube-api-access-9j62v\") pod \"kuadrant-operator-controller-manager-678d87766f-4bvft\" (UID: \"81e4568f-757b-469b-9f75-77565cc0ae98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:44.036012 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:44.035936    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:44.457672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:44.457646    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"]
Oct 31 04:50:44.460428 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:50:44.460396    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e4568f_757b_469b_9f75_77565cc0ae98.slice/crio-5793a673a114928567745fab98be32002a1348ee5f6e96add81d364c54b641bd WatchSource:0}: Error finding container 5793a673a114928567745fab98be32002a1348ee5f6e96add81d364c54b641bd: Status 404 returned error can't find the container with id 5793a673a114928567745fab98be32002a1348ee5f6e96add81d364c54b641bd
Oct 31 04:50:45.362225 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:45.362188    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft" event={"ID":"81e4568f-757b-469b-9f75-77565cc0ae98","Type":"ContainerStarted","Data":"4cd91b66ad9eddf7e10efa5da64878a7d14b49c26e758eb3764dcaa51e3dab6b"}
Oct 31 04:50:45.362686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:45.362252    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:50:45.362686 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:45.362274    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft" event={"ID":"81e4568f-757b-469b-9f75-77565cc0ae98","Type":"ContainerStarted","Data":"5793a673a114928567745fab98be32002a1348ee5f6e96add81d364c54b641bd"}
Oct 31 04:50:45.562518 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:45.562454    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft" podStartSLOduration=2.56243573 podStartE2EDuration="2.56243573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:50:45.515071122 +0000 UTC m=+6195.529632830" watchObservedRunningTime="2026-10-31 04:50:45.56243573 +0000 UTC m=+6195.576997435"
Oct 31 04:50:56.370368 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:50:56.370335    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4bvft"
Oct 31 04:51:09.533841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.533621    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"]
Oct 31 04:51:09.535717 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.535676    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d03b45-79bb-4b18-88d8-471da8dd66de" containerName="istio-proxy"
Oct 31 04:51:09.540260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.540236    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.585501 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.585461    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.585687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.585656    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.585766 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.585688    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgxf\" (UniqueName: \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.587725 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.587701    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"]
Oct 31 04:51:09.630975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.630942    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:51:09.630975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.630963    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:51:09.631180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.630994    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:51:09.634836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.634821    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:51:09.686465 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.686435    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.686648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.686503    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.686648 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.686558    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgxf\" (UniqueName: \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.687142 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.687117    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.689041 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.689023    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.739691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.739661    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgxf\" (UniqueName: \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\") pod \"maas-api-revoke-sub-d576e68b-2vs9m\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:09.779975 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.779944    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:51:09.854908 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:09.854826    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:10.226518 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:10.226488    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"]
Oct 31 04:51:10.229661 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:51:10.229629    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce2fd33_9ff8_46ed_9b10_dff8f4b00d72.slice/crio-fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4 WatchSource:0}: Error finding container fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4: Status 404 returned error can't find the container with id fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4
Oct 31 04:51:10.541780 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:10.541746    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m" event={"ID":"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72","Type":"ContainerStarted","Data":"f41c8921f597a171b75f3aa4b39e84e51a72c682297fc8af619d984a2b345e9d"}
Oct 31 04:51:10.542147 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:10.541799    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m" event={"ID":"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72","Type":"ContainerStarted","Data":"fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4"}
Oct 31 04:51:10.647506 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:10.647452    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m" podStartSLOduration=1.647437199 podStartE2EDuration="1.647437199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:10.645759952 +0000 UTC m=+6220.660321660" watchObservedRunningTime="2026-10-31 04:51:10.647437199 +0000 UTC m=+6220.661998907"
Oct 31 04:51:11.549319 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:11.549282    5831 generic.go:356] "Generic (PLEG): container finished" podID="6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" containerID="f41c8921f597a171b75f3aa4b39e84e51a72c682297fc8af619d984a2b345e9d" exitCode=0
Oct 31 04:51:11.549760 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:11.549364    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m" event={"ID":"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72","Type":"ContainerDied","Data":"f41c8921f597a171b75f3aa4b39e84e51a72c682297fc8af619d984a2b345e9d"}
Oct 31 04:51:12.698175 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.698149    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:12.816123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.816036    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\") pod \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") "
Oct 31 04:51:12.816123 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.816106    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\") pod \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") "
Oct 31 04:51:12.816381 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.816217    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\" (UniqueName: \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\") pod \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\" (UID: \"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72\") "
Oct 31 04:51:12.816504 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.816476    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca" pod "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" (UID: "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:12.818400 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.818369    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca" pod "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" (UID: "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:51:12.818597 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.818576    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf" pod "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" (UID: "6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72"). InnerVolumeSpecName "kube-api-access-nkgxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:12.917229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.917193    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgxf\" (UniqueName: \"kubernetes.io/projected/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-kube-api-access-nkgxf\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:12.917229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.917222    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:12.917229 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:12.917235    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:12.978888 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:51:12.978845    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:51:13.562471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:13.562435    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"
Oct 31 04:51:13.562471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:13.562461    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m" event={"ID":"6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72","Type":"ContainerDied","Data":"fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4"}
Oct 31 04:51:13.562749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:13.562508    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa71ba50ed3e1b2c8bc8cc9b4c6c1f7afded16a029b2d3a2393ad62c71b3d1f4"
Oct 31 04:51:22.207072 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:22.207031    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:51:24.073870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:24.073821    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"]
Oct 31 04:51:24.271562 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:24.259699    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kx6sv"]
Oct 31 04:51:26.134270 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:26.134239    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91484e9-bb28-44ef-a9ff-1efb1e3b19bf" path="/var/lib/kubelet/pods/d91484e9-bb28-44ef-a9ff-1efb1e3b19bf/volumes"
Oct 31 04:51:30.952366 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:30.952329    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"]
Oct 31 04:51:30.952833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:30.952808    5831 cpu_manager.go:414] "containerMap: removing container" podUID="6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" containerName="revoke-keys"
Oct 31 04:51:30.952833 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:30.952823    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" containerName="revoke-keys"
Oct 31 04:51:30.952930 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:30.952919    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" containerName="revoke-keys"
Oct 31 04:51:30.957566 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:30.957516    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.008721 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.008690    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"]
Oct 31 04:51:31.051652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.051616    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:51:31.051652 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.051641    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:51:31.051865 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.051704    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:51:31.052245 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.052230    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:51:31.084455 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.084427    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.084659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.084485    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.084659 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.084606    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8kz\" (UniqueName: \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.185791 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.185754    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.186003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.185836    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.186003 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.185953    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8kz\" (UniqueName: \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.186549 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.186516    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.188580 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.188559    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.239100 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.239034    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8kz\" (UniqueName: \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\") pod \"maas-api-revoke-sub-d2bc93bc-rswhr\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.270921 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.270888    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:31.641980 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.641901    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"]
Oct 31 04:51:31.644740 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:51:31.644708    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12882c4_e79d_44b4_b751_36a6a296a457.slice/crio-097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447 WatchSource:0}: Error finding container 097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447: Status 404 returned error can't find the container with id 097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447
Oct 31 04:51:31.764974 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:31.764936    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr" event={"ID":"e12882c4-e79d-44b4-b751-36a6a296a457","Type":"ContainerStarted","Data":"097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447"}
Oct 31 04:51:32.315466 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:32.315425    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:51:32.772929 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:32.772890    5831 generic.go:356] "Generic (PLEG): container finished" podID="e12882c4-e79d-44b4-b751-36a6a296a457" containerID="e04d5b6349c08204db483f2d6f4e4b393d6ba87666d8bb2d1c5b0fe6e00a2760" exitCode=0
Oct 31 04:51:32.773099 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:32.772957    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr" event={"ID":"e12882c4-e79d-44b4-b751-36a6a296a457","Type":"ContainerDied","Data":"e04d5b6349c08204db483f2d6f4e4b393d6ba87666d8bb2d1c5b0fe6e00a2760"}
Oct 31 04:51:33.921266 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:33.921238    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:34.012849 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.012815    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\") pod \"e12882c4-e79d-44b4-b751-36a6a296a457\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") "
Oct 31 04:51:34.013065 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.012868    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\" (UniqueName: \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\") pod \"e12882c4-e79d-44b4-b751-36a6a296a457\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") "
Oct 31 04:51:34.013065 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.012903    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\") pod \"e12882c4-e79d-44b4-b751-36a6a296a457\" (UID: \"e12882c4-e79d-44b4-b751-36a6a296a457\") "
Oct 31 04:51:34.013388 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.013352    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca" pod "e12882c4-e79d-44b4-b751-36a6a296a457" (UID: "e12882c4-e79d-44b4-b751-36a6a296a457"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:34.015359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.015335    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz" pod "e12882c4-e79d-44b4-b751-36a6a296a457" (UID: "e12882c4-e79d-44b4-b751-36a6a296a457"). InnerVolumeSpecName "kube-api-access-7l8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:34.015359 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.015337    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca" pod "e12882c4-e79d-44b4-b751-36a6a296a457" (UID: "e12882c4-e79d-44b4-b751-36a6a296a457"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:51:34.114420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.114331    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-xks-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:34.114420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.114362    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8kz\" (UniqueName: \"kubernetes.io/projected/e12882c4-e79d-44b4-b751-36a6a296a457-kube-api-access-7l8kz\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:34.114420 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.114378    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12882c4-e79d-44b4-b751-36a6a296a457-maas-api-openshift-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:34.787588 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.787559    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"
Oct 31 04:51:34.787811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.787586    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr" event={"ID":"e12882c4-e79d-44b4-b751-36a6a296a457","Type":"ContainerDied","Data":"097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447"}
Oct 31 04:51:34.787811 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:34.787646    5831 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097694e08f0e706fc3b4bf2dfb3f6ceb7da268bf45af21f0c4bfc645ed266447"
Oct 31 04:51:35.044490 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:35.044393    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"]
Oct 31 04:51:35.049068 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:35.049038    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ld9mp"]
Oct 31 04:51:36.134008 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:36.133975    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f" path="/var/lib/kubelet/pods/7e2b1c32-d02a-4b3d-9fe2-4e37a3cc138f/volumes"
Oct 31 04:51:38.352779 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:38.352746    5831 scope.go:122] "RemoveContainer" containerID="88e07f6cf5af42f003d741f912c2282db78872555af62a63dd17c3314d9940ec"
Oct 31 04:51:40.775310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:40.775271    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:51:42.977463 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:51:42.977422    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:51:43.795383 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.795333    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:51:43.796069 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.796049    5831 cpu_manager.go:414] "containerMap: removing container" podUID="e12882c4-e79d-44b4-b751-36a6a296a457" containerName="revoke-keys"
Oct 31 04:51:43.796130 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.796073    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12882c4-e79d-44b4-b751-36a6a296a457" containerName="revoke-keys"
Oct 31 04:51:43.796220 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.796210    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12882c4-e79d-44b4-b751-36a6a296a457" containerName="revoke-keys"
Oct 31 04:51:43.800070 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.800052    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:43.850726 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.850690    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:51:43.894295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.894263    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:51:43.894295 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.894276    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:51:43.894586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.894467    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:51:43.894586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.894565    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 04:51:43.894586 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.894581    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 04:51:43.908176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.908145    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:43.908320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.908192    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674xp\" (UniqueName: \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:43.908320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.908301    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:43.908431 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:43.908365    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.009350 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.009313    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.009350 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.009352    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674xp\" (UniqueName: \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.009932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.009381    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.009932 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.009420    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.010134 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.010115    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.012024 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.012003    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.012136 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.012118    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.061976 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.061906    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-674xp\" (UniqueName: \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\") pod \"maas-api-997c84955-84mld\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") " pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.117296 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.117262    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:44.487043 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.487018    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:51:44.489496 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:51:44.489467    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ad2eed_bea2_4971_921f_5d4494e2e8ef.slice/crio-00c88c6c2f6abdfaf7b464cd5e0f2664f636bfd9528ed8810ad2b23f01054c2c WatchSource:0}: Error finding container 00c88c6c2f6abdfaf7b464cd5e0f2664f636bfd9528ed8810ad2b23f01054c2c: Status 404 returned error can't find the container with id 00c88c6c2f6abdfaf7b464cd5e0f2664f636bfd9528ed8810ad2b23f01054c2c
Oct 31 04:51:44.490841 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.490821    5831 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:51:44.849768 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:44.849663    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" event={"ID":"17ad2eed-bea2-4971-921f-5d4494e2e8ef","Type":"ContainerStarted","Data":"00c88c6c2f6abdfaf7b464cd5e0f2664f636bfd9528ed8810ad2b23f01054c2c"}
Oct 31 04:51:45.857746 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:45.857634    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" event={"ID":"17ad2eed-bea2-4971-921f-5d4494e2e8ef","Type":"ContainerStarted","Data":"0e24d2af69976bc7a96fb8f9798f4c9a35c78d9e28cae3e9ce7bcef5669aee99"}
Oct 31 04:51:45.858187 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:45.857819    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:45.964289 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:45.964235    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" podStartSLOduration=1.984753434 podStartE2EDuration="2.964220985s" totalImagesPullingTime="979.467551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:45.963468794 +0000 UTC m=+6255.978030503" watchObservedRunningTime="2026-10-31 04:51:45.964220985 +0000 UTC m=+6255.978782693"
Oct 31 04:51:51.881052 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:51.881021    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:51:52.415130 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.415093    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"]
Oct 31 04:51:52.419693 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.419673    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.503749 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.503714    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"]
Oct 31 04:51:52.599764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.599733    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-maas-api-tls\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.599943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.599777    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6bc\" (UniqueName: \"kubernetes.io/projected/9c755368-d8ea-443b-9c17-f56bb7d37bbe-kube-api-access-rl6bc\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.599943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.599859    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-metrics-certs\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.599943 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.599914    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c755368-d8ea-443b-9c17-f56bb7d37bbe-openshift-service-ca\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.701582 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.701508    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-maas-api-tls\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.701783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.701602    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6bc\" (UniqueName: \"kubernetes.io/projected/9c755368-d8ea-443b-9c17-f56bb7d37bbe-kube-api-access-rl6bc\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.701783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.701633    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-metrics-certs\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.701783 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.701682    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c755368-d8ea-443b-9c17-f56bb7d37bbe-openshift-service-ca\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.702495 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.702418    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c755368-d8ea-443b-9c17-f56bb7d37bbe-openshift-service-ca\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.704203 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.704176    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-maas-api-tls\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.704320 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.704239    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c755368-d8ea-443b-9c17-f56bb7d37bbe-metrics-certs\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:52.754621 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:52.754585    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6bc\" (UniqueName: \"kubernetes.io/projected/9c755368-d8ea-443b-9c17-f56bb7d37bbe-kube-api-access-rl6bc\") pod \"maas-api-85868cf77f-tbg8x\" (UID: \"9c755368-d8ea-443b-9c17-f56bb7d37bbe\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:53.033477 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:53.033387    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:53.401439 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:53.401415    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"]
Oct 31 04:51:53.403865 ip-10-0-140-194 kubenswrapper[5831]: W1031 04:51:53.403835    5831 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c755368_d8ea_443b_9c17_f56bb7d37bbe.slice/crio-b48a1f07b2806c83f2a1c4100a4e3879923f78460fec4c4a33d2597222013562 WatchSource:0}: Error finding container b48a1f07b2806c83f2a1c4100a4e3879923f78460fec4c4a33d2597222013562: Status 404 returned error can't find the container with id b48a1f07b2806c83f2a1c4100a4e3879923f78460fec4c4a33d2597222013562
Oct 31 04:51:53.921816 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:53.921776    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x" event={"ID":"9c755368-d8ea-443b-9c17-f56bb7d37bbe","Type":"ContainerStarted","Data":"b48a1f07b2806c83f2a1c4100a4e3879923f78460fec4c4a33d2597222013562"}
Oct 31 04:51:54.931180 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:54.931143    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x" event={"ID":"9c755368-d8ea-443b-9c17-f56bb7d37bbe","Type":"ContainerStarted","Data":"0f1cf60f9770cd64ef631db7c0abb58837bfb69ca0277b022ec5c291ccedb043"}
Oct 31 04:51:54.931709 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:54.931270    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:51:55.039962 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:55.039906    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x" podStartSLOduration=2.118813403 podStartE2EDuration="3.039891673s" totalImagesPullingTime="921.07827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:55.039435084 +0000 UTC m=+6265.053996792" watchObservedRunningTime="2026-10-31 04:51:55.039891673 +0000 UTC m=+6265.054453381"
Oct 31 04:51:56.350632 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:51:56.350595    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:52:00.945200 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:00.945161    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85868cf77f-tbg8x"
Oct 31 04:52:01.121508 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:01.121468    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:52:01.122092 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:01.122038    5831 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerName="maas-api" containerID="cri-o://0e24d2af69976bc7a96fb8f9798f4c9a35c78d9e28cae3e9ce7bcef5669aee99" gracePeriod=30
Oct 31 04:52:02.050856 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.050816    5831 generic.go:356] "Generic (PLEG): container finished" podID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerID="0e24d2af69976bc7a96fb8f9798f4c9a35c78d9e28cae3e9ce7bcef5669aee99" exitCode=0
Oct 31 04:52:02.051671 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.051641    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" event={"ID":"17ad2eed-bea2-4971-921f-5d4494e2e8ef","Type":"ContainerDied","Data":"0e24d2af69976bc7a96fb8f9798f4c9a35c78d9e28cae3e9ce7bcef5669aee99"}
Oct 31 04:52:02.052048 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.051952    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:52:02.078116 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.078078    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\") pod \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") "
Oct 31 04:52:02.078888 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.078863    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\") pod \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") "
Oct 31 04:52:02.079491 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.079472    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\" (UniqueName: \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\") pod \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") "
Oct 31 04:52:02.080029 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.080010    5831 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\") pod \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\" (UID: \"17ad2eed-bea2-4971-921f-5d4494e2e8ef\") "
Oct 31 04:52:02.080871 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.078744    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca" pod "17ad2eed-bea2-4971-921f-5d4494e2e8ef" (UID: "17ad2eed-bea2-4971-921f-5d4494e2e8ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:52:02.096206 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.096152    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp" pod "17ad2eed-bea2-4971-921f-5d4494e2e8ef" (UID: "17ad2eed-bea2-4971-921f-5d4494e2e8ef"). InnerVolumeSpecName "kube-api-access-674xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:52:02.096515 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.096477    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls" pod "17ad2eed-bea2-4971-921f-5d4494e2e8ef" (UID: "17ad2eed-bea2-4971-921f-5d4494e2e8ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:52:02.096764 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.096725    5831 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs" pod "17ad2eed-bea2-4971-921f-5d4494e2e8ef" (UID: "17ad2eed-bea2-4971-921f-5d4494e2e8ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:52:02.190109 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.190030    5831 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ad2eed-bea2-4971-921f-5d4494e2e8ef-openshift-service-ca\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:02.190327 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.190309    5831 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-maas-api-tls\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:02.190443 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.190428    5831 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-674xp\" (UniqueName: \"kubernetes.io/projected/17ad2eed-bea2-4971-921f-5d4494e2e8ef-kube-api-access-674xp\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:02.190576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.190561    5831 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ad2eed-bea2-4971-921f-5d4494e2e8ef-metrics-certs\") on node \"ip-10-0-140-194.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:52:02.608616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.595957    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:52:02.608616 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:02.596028    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:52:03.033558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.032823    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:52:03.033558 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.032922    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:52:03.082876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.078693    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" event={"ID":"17ad2eed-bea2-4971-921f-5d4494e2e8ef","Type":"ContainerDied","Data":"00c88c6c2f6abdfaf7b464cd5e0f2664f636bfd9528ed8810ad2b23f01054c2c"}
Oct 31 04:52:03.082876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.078780    5831 scope.go:122] "RemoveContainer" containerID="0e24d2af69976bc7a96fb8f9798f4c9a35c78d9e28cae3e9ce7bcef5669aee99"
Oct 31 04:52:03.082876 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.078952    5831 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld"
Oct 31 04:52:03.089101 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.089052    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:52:03.089400 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.089355    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:52:03.267645 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.267603    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:52:03.274038 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:03.273998    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-84mld"]
Oct 31 04:52:04.134775 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:04.134743    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" path="/var/lib/kubelet/pods/17ad2eed-bea2-4971-921f-5d4494e2e8ef/volumes"
Oct 31 04:52:04.882820 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:04.882773    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-997c84955-84mld" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:52:12.978082 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:52:12.978043    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:52:38.377472 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.377437    5831 scope.go:122] "RemoveContainer" containerID="96cd2e87b8dc11c77c2782d44c336481fe19e2a3abeacec32602a819f73aa5eb"
Oct 31 04:52:38.388782 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.388755    5831 scope.go:122] "RemoveContainer" containerID="337627438ff607d711332176b3abf302f780d70c113e4c9173a426312510081c"
Oct 31 04:52:38.397755 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.397722    5831 scope.go:122] "RemoveContainer" containerID="c0698f0902b7a552ae3eeb4a654129f2b070d3f30a0a3d0ea8ad256c3c31867b"
Oct 31 04:52:38.967302 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.967266    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:52:38.979079 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.979051    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:52:38.979281 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.979127    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:52:38.983338 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.983313    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:52:38.990960 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.990934    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:52:38.995011 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:38.994994    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:52:39.876576 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:39.876541    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:52:42.978055 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:52:42.978014    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:52:52.303118 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:52:52.303081    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:53:12.977431 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:53:12.977377    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:53:16.562645 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:16.562606    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:53:26.045355 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:26.045304    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"]
Oct 31 04:53:26.116990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:26.116950    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e39e43-ww29p"]
Oct 31 04:53:26.141554 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:26.141149    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7aeb125-6aec-4ae8-b671-ac822519a42d" path="/var/lib/kubelet/pods/e7aeb125-6aec-4ae8-b671-ac822519a42d/volumes"
Oct 31 04:53:29.300710 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:29.300676    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:53:32.598172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:32.597638    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ddzvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:53:32.598172 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:32.597707    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ddzvr" podUID="226eca67-9a93-4f42-970d-cc145e14826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:53:38.435471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:38.435436    5831 scope.go:122] "RemoveContainer" containerID="d72b157ef0adfb14fed2928c09d1bfe3dbab72abdec7f9e2cce5aadb90f9b3b5"
Oct 31 04:53:42.977928 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:53:42.977887    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:53:58.272953 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:53:58.272873    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:54:08.958074 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:08.958038    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:54:12.977766 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:54:12.977716    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:54:18.035846 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:18.035801    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:54:18.038702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:18.038660    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:54:18.105695 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:18.105657    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:54:18.105991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:18.105957    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:54:21.852486 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:21.852453    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:54:25.047890 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:25.047845    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"]
Oct 31 04:54:25.115556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:25.109307    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g2ghb"]
Oct 31 04:54:26.135156 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:26.135125    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36369f2-8c5b-40d3-b6c3-806ac47cf65f" path="/var/lib/kubelet/pods/e36369f2-8c5b-40d3-b6c3-806ac47cf65f/volumes"
Oct 31 04:54:31.589807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:31.589773    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:54:38.459049 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:38.459013    5831 scope.go:122] "RemoveContainer" containerID="b31d5fb09e51a5b840543809ad870b2346ea2039c84dc0dc950ca26c1b147a7d"
Oct 31 04:54:42.978362 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:54:42.978320    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:54:56.775836 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:54:56.775785    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:55:07.483697 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:07.483661    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:55:12.992134 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:55:12.992083    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:55:17.456473 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:17.456437    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:55:33.043887 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:33.043673    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:55:33.044734 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:33.044699    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vrf" podUID="bc914aa8-e974-404a-b1e4-3d5c05828693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:55:33.085688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:33.085642    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nhf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:55:33.086021 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:33.085988    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nhf4g" podUID="acc00eba-a262-434c-8c79-d887a3cb9697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:55:36.964667 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:36.964631    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:55:38.508556 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:38.507846    5831 scope.go:122] "RemoveContainer" containerID="e4d4dd2c7a54eaf13793c56c2d8d368251a67207e29d40654bc525b6344f2d1e"
Oct 31 04:55:42.982128 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:55:42.982090    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:55:58.065915 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:55:58.065880    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:56:12.977728 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:56:12.977683    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:56:13.043990 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:13.043957    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"]
Oct 31 04:56:13.089754 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:13.089711    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2vs9m"]
Oct 31 04:56:14.134446 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:14.134411    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72" path="/var/lib/kubelet/pods/6ce2fd33-9ff8-46ed-9b10-dff8f4b00d72/volumes"
Oct 31 04:56:20.557557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:20.556720    5831 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lqmhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:56:20.557557 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:20.556810    5831 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lqmhm" podUID="e81cdb3c-2ad9-4df5-af12-8d20de42ce47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:56:35.044001 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:35.043962    5831 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"]
Oct 31 04:56:35.090349 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:35.090316    5831 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rswhr"]
Oct 31 04:56:36.133655 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:56:36.133624    5831 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12882c4-e79d-44b4-b751-36a6a296a457" path="/var/lib/kubelet/pods/e12882c4-e79d-44b4-b751-36a6a296a457/volumes"
Oct 31 04:56:43.009873 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:56:43.009642    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:57:12.990683 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:57:12.990558    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:57:38.602310 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:38.602275    5831 scope.go:122] "RemoveContainer" containerID="e04d5b6349c08204db483f2d6f4e4b393d6ba87666d8bb2d1c5b0fe6e00a2760"
Oct 31 04:57:38.612010 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:38.611985    5831 scope.go:122] "RemoveContainer" containerID="f41c8921f597a171b75f3aa4b39e84e51a72c682297fc8af619d984a2b345e9d"
Oct 31 04:57:39.010294 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.010262    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:57:39.020702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.020676    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:57:39.024250 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.024231    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:57:39.027910 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.027888    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gcndh_a3139e21-cc0c-4910-8a21-e99bc0e5662e/extract/0.log"
Oct 31 04:57:39.036912 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.036896    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tkdn9_1b558d27-fb04-4431-88a3-22d27045dacb/console-operator/2.log"
Oct 31 04:57:39.040232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:39.040216    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9hwd_f8dcdbcd-d8e5-4400-880b-c89f3362af1b/ovn-acl-logging/0.log"
Oct 31 04:57:42.978601 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:57:42.978561    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:57:55.363434 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:57:55.363395    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7phrx"]
Oct 31 04:58:12.979151 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:58:12.979101    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:58:15.062488 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:15.062456    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d8444996-2rgdr_12c09a2b-78b4-4175-826f-781805aee163/authorino/0.log"
Oct 31 04:58:27.601347 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:27.601319    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:58:28.209664 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:28.209584    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:58:30.192624 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:30.192583    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85868cf77f-tbg8x_9c755368-d8ea-443b-9c17-f56bb7d37bbe/maas-api/0.log"
Oct 31 04:58:35.894702 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:35.894675    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d8444996-2rgdr_12c09a2b-78b4-4175-826f-781805aee163/authorino/0.log"
Oct 31 04:58:36.314691 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:36.314658    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dwqhg_b13d7e68-3328-4ef1-9449-fddfcce22cea/manager/0.log"
Oct 31 04:58:36.545064 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:36.545034    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mxvkt_7da6fd01-0c72-4691-b956-6807e32461c1/manager/0.log"
Oct 31 04:58:37.092278 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:37.092239    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4bvft_81e4568f-757b-469b-9f75-77565cc0ae98/manager/0.log"
Oct 31 04:58:37.406594 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:37.406512    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7phrx_2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc/limitador/0.log"
Oct 31 04:58:38.627957 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:38.627926    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5slg5v_8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e/istio-proxy/0.log"
Oct 31 04:58:38.855737 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:38.855698    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkrml_8d43e0c8-cc75-4e72-8308-8972e3a3e1f8/discovery/0.log"
Oct 31 04:58:40.430711 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:40.430674    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ddzvr_226eca67-9a93-4f42-970d-cc145e14826c/istio-proxy/0.log"
Oct 31 04:58:40.665583 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:40.665518    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-h4vrf_bc914aa8-e974-404a-b1e4-3d5c05828693/istio-proxy/0.log"
Oct 31 04:58:40.900096 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:40.900008    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lqmhm_e81cdb3c-2ad9-4df5-af12-8d20de42ce47/istio-proxy/0.log"
Oct 31 04:58:41.358681 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:41.358643    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nhf4g_acc00eba-a262-434c-8c79-d887a3cb9697/istio-proxy/0.log"
Oct 31 04:58:41.812323 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:41.812292    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph_c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3/istio-proxy/0.log"
Oct 31 04:58:42.470838 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:42.470808    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6489c4dbb7-6rbbc_d608bf35-2830-4486-a05f-ac59ddb3c8e6/router/0.log"
Oct 31 04:58:42.977571 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:58:42.977510    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:58:44.798991 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:58:44.798959    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hv9rw_3c71e606-ef12-4b55-8e2a-1019c02014f5/main/0.log"
Oct 31 04:59:03.946723 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:03.946684    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bj5qd_5feec31a-0c22-428a-9d9a-3c16b3cd6097/global-pull-secret-syncer/0.log"
Oct 31 04:59:05.672232 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:05.672203    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-675lv_bfe907fb-0ddb-48d8-8a02-8ee398870770/connection-checker/0.log"
Oct 31 04:59:06.077626 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:06.077598    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jjkm4_71bad7a7-3b59-4e50-96d9-59dcd2ec790c/connection-checker/0.log"
Oct 31 04:59:06.484847 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:06.484821    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qqbzc_ead07c5f-c2fb-42de-9254-059fab1e0ab7/connection-checker/0.log"
Oct 31 04:59:07.388590 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:07.388555    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhzmz_984c8e10-fc2b-4012-9b5b-78b9e3770178/konnectivity-agent/0.log"
Oct 31 04:59:08.213767 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:08.213725    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:08.358914 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:08.358877    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:08.973186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:08.973151    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-194.us-west-2.compute.internal_d102acf3d8823a0d171d30a06e49c39c/haproxy/0.log"
Oct 31 04:59:09.363687 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:09.363595    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:09.580363 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:09.580332    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:12.871260 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:12.871227    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d8444996-2rgdr_12c09a2b-78b4-4175-826f-781805aee163/authorino/0.log"
Oct 31 04:59:12.978255 ip-10-0-140-194 kubenswrapper[5831]: E1031 04:59:12.978220    5831 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-svcxz" containerName="extractor"
Oct 31 04:59:14.108454 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:14.108422    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dwqhg_b13d7e68-3328-4ef1-9449-fddfcce22cea/manager/0.log"
Oct 31 04:59:14.512039 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:14.512008    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mxvkt_7da6fd01-0c72-4691-b956-6807e32461c1/manager/0.log"
Oct 31 04:59:15.175016 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.174979    5831 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"]
Oct 31 04:59:15.175591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.175478    5831 cpu_manager.go:414] "containerMap: removing container" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerName="maas-api"
Oct 31 04:59:15.175591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.175490    5831 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerName="maas-api"
Oct 31 04:59:15.175591 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.175589    5831 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ad2eed-bea2-4971-921f-5d4494e2e8ef" containerName="maas-api"
Oct 31 04:59:15.178771 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.178754    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.222857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.222826    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-podres\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.222857 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.222857    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-proc\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.223112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.222954    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-lib-modules\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.223112 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.222996    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-sys\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.223199 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.223167    5831 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtdq\" (UniqueName: \"kubernetes.io/projected/c95a9fc5-9e33-433c-83fb-6d45a7122fde-kube-api-access-8qtdq\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.228506 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.228486    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"openshift-service-ca.crt\""
Oct 31 04:59:15.274629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.274598    5831 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"kube-root-ca.crt\""
Oct 31 04:59:15.274629 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.274608    5831 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m7rwt\"/\"default-dockercfg-74bbv\""
Oct 31 04:59:15.278578 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.278551    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"]
Oct 31 04:59:15.323870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.323840    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-podres\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.323870 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.323879    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-proc\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.323922    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-lib-modules\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.323948    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-sys\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.323985    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-podres\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.324018    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-proc\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.324020    5831 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtdq\" (UniqueName: \"kubernetes.io/projected/c95a9fc5-9e33-433c-83fb-6d45a7122fde-kube-api-access-8qtdq\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324154 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.324051    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-sys\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.324481 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.324199    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c95a9fc5-9e33-433c-83fb-6d45a7122fde-lib-modules\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.376684 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.376653    5831 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtdq\" (UniqueName: \"kubernetes.io/projected/c95a9fc5-9e33-433c-83fb-6d45a7122fde-kube-api-access-8qtdq\") pod \"perf-node-gather-daemonset-dqvw4\" (UID: \"c95a9fc5-9e33-433c-83fb-6d45a7122fde\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.387197 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.387171    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4bvft_81e4568f-757b-469b-9f75-77565cc0ae98/manager/0.log"
Oct 31 04:59:15.491772 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.491746    5831 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:15.704471 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.704433    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:15.729076 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.729046    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7phrx_2cae7632-d1fb-4de5-ba4c-d1f12f0d55fc/limitador/0.log"
Oct 31 04:59:15.861389 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.861294    5831 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"]
Oct 31 04:59:15.867164 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:15.867140    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:16.672357 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.672323    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:16.721672 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.721640    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4" event={"ID":"c95a9fc5-9e33-433c-83fb-6d45a7122fde","Type":"ContainerStarted","Data":"49dbf4ce58e14d303da5e17281d5d64796dd2db78e608570cde0228f0fc8016c"}
Oct 31 04:59:16.721895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.721703    5831 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4" event={"ID":"c95a9fc5-9e33-433c-83fb-6d45a7122fde","Type":"ContainerStarted","Data":"204ea1f499179c8ffd708dc31344ccaa9ad1e1022e032dc2bbabe2f0b91e09ff"}
Oct 31 04:59:16.721895 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.721726    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:16.831000 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.830906    5831 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4" podStartSLOduration=1.830853257 podStartE2EDuration="1.830853257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:16.826989152 +0000 UTC m=+6706.841550860" watchObservedRunningTime="2026-10-31 04:59:16.830853257 +0000 UTC m=+6706.845414977"
Oct 31 04:59:16.862773 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:16.862737    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:21.767183 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:21.767151    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xv98j_ec33cd8c-1d9c-437b-8742-661d57128951/cluster-monitoring-operator/0.log"
Oct 31 04:59:22.503449 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:22.503414    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:22.614014 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:22.613977    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7pq8r_d7d376d2-9269-4725-9b05-bbad81da4a84/manager/0.log"
Oct 31 04:59:22.740707 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:22.740677    5831 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-dqvw4"
Oct 31 04:59:23.761650 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:23.761617    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:23.961395 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:23.961360    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c946769f6-6pgn5_90ef7101-8986-453d-aa53-e1a5c823b6cb/manager/0.log"
Oct 31 04:59:25.375807 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:25.375771    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvwqp_79933f85-942a-45b0-86f2-c22a0ea57599/node-exporter/0.log"
Oct 31 04:59:25.781186 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:25.781157    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvwqp_79933f85-942a-45b0-86f2-c22a0ea57599/kube-rbac-proxy/0.log"
Oct 31 04:59:26.188882 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:26.188859    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvwqp_79933f85-942a-45b0-86f2-c22a0ea57599/init-textfile/0.log"
Oct 31 04:59:29.413429 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:29.413384    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5slg5v_8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e/istio-proxy/0.log"
Oct 31 04:59:29.714250 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:29.714219    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5slg5v_8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e/istio-proxy/0.log"
Oct 31 04:59:34.153492 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:34.153442    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkrml_8d43e0c8-cc75-4e72-8308-8972e3a3e1f8/discovery/0.log"
Oct 31 04:59:34.466628 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:34.466600    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkrml_8d43e0c8-cc75-4e72-8308-8972e3a3e1f8/discovery/0.log"
Oct 31 04:59:35.513061 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:35.513028    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ddzvr_226eca67-9a93-4f42-970d-cc145e14826c/istio-proxy/0.log"
Oct 31 04:59:35.723409 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:35.723376    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-h4vrf_bc914aa8-e974-404a-b1e4-3d5c05828693/istio-proxy/0.log"
Oct 31 04:59:35.818135 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:35.818039    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ddzvr_226eca67-9a93-4f42-970d-cc145e14826c/istio-proxy/0.log"
Oct 31 04:59:35.930517 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:35.930488    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lqmhm_e81cdb3c-2ad9-4df5-af12-8d20de42ce47/istio-proxy/0.log"
Oct 31 04:59:36.026778 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:36.026745    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-h4vrf_bc914aa8-e974-404a-b1e4-3d5c05828693/istio-proxy/0.log"
Oct 31 04:59:36.237193 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:36.237159    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lqmhm_e81cdb3c-2ad9-4df5-af12-8d20de42ce47/istio-proxy/0.log"
Oct 31 04:59:36.346230 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:36.346195    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nhf4g_acc00eba-a262-434c-8c79-d887a3cb9697/istio-proxy/0.log"
Oct 31 04:59:36.655114 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:36.655034    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nhf4g_acc00eba-a262-434c-8c79-d887a3cb9697/istio-proxy/0.log"
Oct 31 04:59:36.757255 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:36.757218    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph_c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3/istio-proxy/0.log"
Oct 31 04:59:37.066644 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:37.066613    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-vd5ph_c9ecaedf-c39e-48e3-be9c-62ceb36ec1e3/istio-proxy/0.log"
Oct 31 04:59:37.404176 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:37.404084    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6489c4dbb7-6rbbc_d608bf35-2830-4486-a05f-ac59ddb3c8e6/router/0.log"
Oct 31 04:59:37.529688 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:37.529648    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5slg5v_8d8bf850-49a7-4b6c-ae2f-c7daafc7c61e/istio-proxy/0.log"
Oct 31 04:59:37.704457 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:37.704421    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6489c4dbb7-6rbbc_d608bf35-2830-4486-a05f-ac59ddb3c8e6/router/0.log"
Oct 31 04:59:39.172316 ip-10-0-140-194 kubenswrapper[5831]: I1031 04:59:39.172275    5831 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-dbpk4_01f13667-aa82-4f8e-b860-abcfb602fc4c/networking-console-plugin/0.log"